Podcasts about regularization

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Measuring Learned Optimization in Small Transformer Models, published by Jonathan Bostock on April 8, 2024 on The AI Alignment Forum. This is original, independent research carried out in March and April of 2024. The degree to which a a policy optimizes the future can be quantified mathematically. A set of of very small transformer models were pretrained to predict the next token in a mathematical sequence, then subjected to reinforcement learning finetuning. The optimizing power of each model can be predicted with high accuracy based on each model's score on its own RL task. By comparing predictions of optimization based on scores on each different RL task, a model's original reinforcement objective can be identified. A related measure for impact can also be derived mathematically, and given a theoretical lower bound based on RL score. This gives further information about model behavior, and allows for the same analysis as the measure of optimization. I also investigate the possibility of getting models to self-evaluate optimization and impact, with limited success. Methods Pretraining on Sequence Prediction I defined a simple mathematical sequence defined by the following stochastic recurrence relation. This produces a pseudo-random but (to 98%) predictable sequence, alternating between elements of {0,...,7} on even values of t and {8,...,15} on odd values of t. st=(((16i=1(sti+1)mod17)mod8) with probability 98% {0,...,7} with probabiltiy 2%)+8(tmod2) I then trained a small encoder-only transformer model to predict the next element in the sequence given the previous 20 elements of the sequence. This was followed by a reinforcement-learning phase in which the transformer was used to generate the next token on odd values of n only, and the recurrence relation was used to generate the value of st+1. If st+1 was in {0,2,4,6}, this was used as a "successful" example to reinforce the model. I used a temperature of 1 when generating these sequences to introduce some randomness, but the temperature was reduced to 0 during evaluations and when calculating optimization. A small amount of "maintenance" training (much lower learning rate) was used during this phase to ensure that model performance on the predictive tasks for even values of t was maintained. Without this I saw rapid loss of performance on the "maintenance" dataset. I also found that I was unable to include "unsuccessful" examples (i.e. where st+1{0,2,4,6}) with even a tiny negative learning rate, as this caused worsened performance at all tasks. Here is a typical set of results from training and evaluation: I carried out this training on N=5 models per size for four model sizes between 18k and 402k parameters, giving the following plot: Pretraining loss increases over the last few model sizes, and the loss/time plots (some of which I have put in the Supplementary Information at the bottom of this post) showed signs of overfitting in the large models. Regularization was employed during training (0.01 weight decay in an AdamW optimizer, 10% dropout rate for neurons) so perhaps a larger dataset size is required to totally avoid this. I then repeated the RL phase twice, once with st+1{0,4} being reinforced, (ngood = 2) and once with st+1{0,1,2,4,5,6} being reinforced (ngood = 6). Here is a plot of success rate against model size across all three conditions. This plot shows mean standard error. In all cases model performance is a lot better than chance, and increases with model size. Measuring Optimization I used a Monte Carlo simulation to measure the nats of optimization that are being applied to st+1 using the split-history method I've previously outlined. This involves taking the difference in entropy between two distributions: The algorithm in practice is this: Take a bunch of sequence examples from the testing...

Before the Lieber Code and Geneva Conventions came a treaty between the Spanish Empire and Simon Bolivar's revolutionary forces in Colombia and Venezuela. The 1820 Treaty for the Regularization of War aimed at reducing the unnecessary suffering of both soldiers and civilians affected by armed conflict and occupation across a broader spectrum than any previous international agreements. However, despite the significance of such a development in international law, the treaty fell into relative obscurity after the Colombian War of Independence until being slowly reintroduced throughout the 20th century. In this post, graduate student Jacob Coffelt from the University of Padua explores what can be considered the birth of international humanitarian law in Latin America as well as the effects colonialism has had on its legacy. Using both historical and contemporary sources, he argues that the codification of modern principles of international humanitarian law had occurred decades prior to what is traditionally suggested.

Vision Transformers (ViT) have been shown to attain highly competitive performance for a wide range of vision applications, such as image classification, object detection and semantic image segmentation. In comparison to convolutional neural networks, the Vision Transformer's weaker inductive bias is generally found to cause an increased reliance on model regularization or data augmentation ("AugReg"for short) when training on smaller training datasets. We conduct a systematic empirical study in order to better understand the interplay between the amount of training data, AugReg, model size and compute budget. As one result of this study we find that the combination of increased compute and AugReg can yield models with the same performance as models trained on an order of magnitude more training data: we train ViT models of various sizes on the public ImageNet-21k dataset which either match or outperform their counterparts trained on the larger, but not publicly available JFT-300M dataset. 2021: A. Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, Lucas Beyer https://arxiv.org/pdf/2106.10270v1.pdf

In today's episode Greg and Patrick talk about regularization, which includes ridge, LASSO, and elastic net procedures for variable selection within the general linear model and beyond. Along the way they also mention Bowdlerizing, The Family Shakespeare, disturbance in the force, McNeish on his bike, Spandex, C'mon guys wait up, the altar of unbiasedness, Curranizing, shooting arrows, stepwise goat rodeo, volume knobs, Hancockizing, always angry, getting slapped, betting a chicken, mission from God, hypothetico-deductive porpoising, and letting go of truth (which you can't handle anyway).Stay in contact with Quantitude! Twitter: @quantitudepod Web page: quantitudepod.org Merch: redbubble.com

The UN special rapporteur on slavery paid an official visit to Canada in late August to assess the government's efforts to prevent and address contemporary forms of slavery. After spending two weeks in Ottawa, Toronto, Vancouver and two other Canadian cities, Tomoya Obokata identified a number of groups in Canada as vulnerable to slavery, including migrant workers brought in through the Temporary Foreign Workers Programme. We talk with Syed Hussan of Migrant Workers Alliance for Change.

Link to bioRxiv paper: http://biorxiv.org/cgi/content/short/2023.04.15.537017v1?rss=1 Authors: Vallarino, E., Hincapie, A. S., Jerbi, K., Leahyf, R., Pascarella, A., Sorrentino, A., Sommariva, S. Abstract: The accurate characterization of cortical functional connectivity from Magnetoencephalography (MEG) data remains a challenging problem due to the subjective nature of the analysis, which requires several decisions at each step of the analysis pipeline, such as the choice of a source estimation algorithm, a connectivity metric and a cortical parcellation, to name but a few. Recent studies have emphasized the importance of selecting the regularization parameter in minimum norm estimates with caution, as variations in its value can result in significant differences in connectivity estimates. In particular, the amount of regularization that is optimal for MEG source estimation can actually be suboptimal for coherence-based MEG connectivity analysis. In this study, we expand upon previous work by examining a broader range of commonly used connectivity metrics, including the imaginary part of coherence, corrected imaginary part of Phase Locking Value, and weighted Phase Lag Index, within a larger and more realistic simulation scenario. Our results show that the best estimate of connectivity is achieved using a regularization parameter that is 1 or 2 orders of magnitude smaller than the one that yields the best source estimation. This remarkable difference may imply that previous work assessing source-space connectivity using minimum-norm may have benefited from using less regularization, as this may have helped reduce false positives. Importantly, we provide the code for MEG data simulation and analysis, offering the research community a valuable open source tool for informed selections of the regularization parameter when using minimum-norm for source space connectivity analyses. Copy rights belong to original authors. Visit the link for more info Podcast created by Paper Player, LLC

InfosecTrain hosts a live event entitled ‘Data Science Fast Track Course' with certified expert ‘NAWAJ'. Data Science is not the future anymore, it is rather the present. This masterclass would be extremely beneficial to anyone interested in pursuing a career in Data Science. It will be delivered by a domain expert with extensive industry experience. With our instructors who are specialists in their disciplines, we hold a global reputation. Attending this webinar will benefit you in a variety of ways. Thank you for Listening this Podcast, For more details or free demo with our expert write into us at sales@infosectrain.com ➡️ Agenda

what is IOD, CC & OC  The Intimation of Disapproval (IOD) is only an approval of the civil plans. Review of the structural plans is done in parallel with the NOC process. No approval to this plan is required from the Municipal Corporation but copies are required to be submitted.  IOD   is the first permit considered for construction. In Mumbai An IOD intimation of Disapproval is a letter issued by the Municipal Corporation of Greater Mumbai (MCGM)  which is  issued to a Builder or Developer of a Property who proposes to undertake some construction work on an existing or new building.  The permitting authority would be BMC/SRA/MMRDA/MHADA/MIDC/ BPT   as per their jurisdiction  Completion Certificate or CC is given in 2 stages. Plinth Commencement certificate CC for start of Work  full CC for project ( which can also be taken as Part CC) It is only after a CC is issued to a project that it becomes ready for possession for home buyers. Whereas, OC stands for Occupation Certificate which states the construction had been completed as per the plan.  An occupation certificate or OC, on the other hand, is a certificate stating that the project has been built in accordance with all construction norms, building bye-laws, etc. sometimes OC comes with BCC Building Completion Certificate or is given seperately Should I buy property without OC? Note that without OC, the building can be demolished anytime. Basic sanitation facilities can be revoked. Govt can send you a eviction notice. Regularization may be allowed on the payment of penalty by the builder, but imagine if the penalty is passed to the buyer I.e, owner..in that case you will suffer.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Gradient surfing: the hidden role of regularization, published by Jesse Hoogland on February 6, 2023 on The AI Alignment Forum. Produced as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort In a previous post, I demonstrated that Brownian motion near singularities defies our expectations from "regular" physics. Singularities trap random motion and take up more of the equilibrium distribution than you'd expect from the Gibbs measure. In the computational probability community, this is a well-known pathology. Sampling techniques like Hamiltonian Monte Carlo get stuck in corners, and this is something to avoid. You typically don't want biased estimates of the distribution you're trying to sample. In deep learning, I argued, this behavior might be less a bug than a feature. The claim of singular learning theory is that models near singularities have lower effective dimensionality. From Occam's razor, we know that simpler models generalize better, so if the dynamics of SGD get stuck at singularities, it would suggest an explanation (at least in part) for why SGD works: the geometry of the loss landscape biases your optimizer towards good solutions. This is not a particularly novel claim. Similar versions of the claim been made before by Mingard et al. and Valle Pérez et al.. But from what I can tell, the proposed mechanism, of singularity "stickiness", is quite different. Moreover, it offers a new possible explanation for the role of regularization. If exploring the set of points with minimum training loss is enough to get to generalization, then perhaps the role of regularizer is not just to privilege "simpler" functions but also to make exploration possible. In the absence of regularization, SGD can't easily move between points of equal loss. When it reaches the bottom of a valley, it's pretty much stuck. Adding a term like weight decay breaks this invariance. It frees the neural network to surf the loss basin, so it can accidentally stumble across better generalizing solutions. So could we improve generalization by exploring the bottom of the loss basin in other ways — without regularization or even without SGD? Could we, for example, get a model to grok through random drift? .No. We can't. That is to say I haven't succeeded yet. Still, in the spirit of "null results are results", let me share the toy model that motivated this hypothesis and the experiments that have (as of yet) failed to confirm it. The inspiration: a toy model First, let's take a look at the model that inspired the hypothesis. Let's begin by modifying the example of the previous post to include an optional regularization term controlled by λ: We deliberately center the regularization away from the origin at c=(−1,−1) so it doesn't already privilege the singularity at the origin. Now, instead of viewing U(x) as a potential and exploring it with Brownian motion, we'll treat it as a loss function and use stochastic gradient descent to optimize for x. We'll start our optimizer at a uniformly sampled random point in this region and take T=100 steps down the gradient (with optional momentum controlled by β). After each gradient step, we'll inject a bit of Gaussian noise to simulate the "stochasticity." Altogether, the update rule for x is as follows: with momentum updated according to: and noise given by, If we sample the final obtained position, x(T) over independent initializations, then, in the absence of regularization and in the presence of a small noise term, we'll get a distribution that looks like the figure on the left. Unlike the case of random motion, the singularity at the origin is now repulsive. Good luck finding those simple solutions now. However, as soon as we turn on the regularization (middle figure) or increase the noise term (figure on the right), the singulari...

In today's episode, we'll be discussing regularization, an important technique used to prevent overfitting in machine learning models. If you find these episodes helpful, please consider supporting us on Patreon at patreon.com/user?u=84843123. Your support will help us continue to produce these episodes and improve the show. Thanks for listening!

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Take 5: Another problem with natural abstractions is laziness., published by Charlie Steiner on December 6, 2022 on The AI Alignment Forum. As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day for 25 days. Or until I run out of hot takes. Soundtrack. Natural abstractions are patterns in the environment that are so convenient and so useful that most right-thinking agents will learn to take advantage of them. But what if humans and modern ML are too lazy to be right-thinking? One way of framing this point is in terms of gradient starvation. The reason neural networks don't explore all possible abstractions (aside from the expense) is that once they find the first way of solving a problem, they don't really have an incentive to find a second way - it doesn't give them a higher score, so they don't. When gradient starvation is strong, it means the loss landscape has a lot of local minima that the agent can roll into, that aren't easily connected to the global minimum, and so what abstractions the network ends up using will depend strongly on the initial conditions. Regularization and exploration can help ameliorate this problem, but often come with catastrophic forgetting - if a neural net finds a strictly better way to solve the problem it's faced with, it might forget all about the previous way When we imagine a right-thinking agent that learns natural abstractions, we often imagine something that's intrinsically motivated to learn lots of different ways of solving a problem, and that doesn't erase its memory of interesting methods just because they're not on the Pareto frontier. So that's what I mean by "lazy"/"not lazy", here. Neural networks, or humans, are lazy if they're parochial in solution-space, doing local search in a way that sees them get stuck in what a less-lazy optimizer might consider to be ruts. It's important to note that laziness is not an unambiguously bad property. First, it's usually more efficient. Second, maybe we don't want our neural net to actually search through the weird and adversarial parts of parameter-space, and local search prevents it from doing so. Alex Turner et al. have recently been making arguments like this fairly forcefully. Still, we don't want maximal laziness, especially not if we want to find natural abstractions like the various meanings of "human values." I might be attacking a strawman or a bailey here, I'm not totally sure. I've been using "natural abstraction" here as if it just means an abstraction that would be useful for a wide variety of agents to have in their toolbox. But we might also use "natural abstractions" to denote the vital abstractions, those that aren't merely nice to have, but that you literally can't complete certain tasks without using. In that second sense, neural networks are always highly incentivized to learn relevant natural abstractions, and you can easily tell when they do so by measuring their loss. But as per yesterday, there are often multiple similarly-powerful ways to model the world, in particular when modeling humans and human values. There might be hard core vital abstractions for various human-interaction tasks, but I suspect they're abstractions like "discrete object," not anything nearly so far into the leaves of the tree as "human values." And when I see informal speculation about natural abstractions it usually strikes me as thinking about the less strict "useful for most agents" abstractions. Ultimately, I expect laziness to cause both artificial neural nets and humans to miss out on some sizeable fraction of abstractions that most agents would find useful. What to do? There are options: Build an AI that isn't lazy. But laziness is useful, and anyhow maybe we don't want an AI to explore all the extrema. So build an AI t...

Regularization of Single Field Inflation Models by Josh Hoffmann et al. on Friday 25 November There are many single field inflationary models that are consistent with the recent Planck 2018 measurements of the spectral index $n_s$ and tensor-to-scalar ratio $r$. Despite good agreement with observational data some of these models suffer from having unregularized potentials which would produce a collapsing universe shortly after the end of inflation. In this paper we show that how one chooses to correct the behaviour potential towards the end of inflation can have a significant effect on the inflationary predictions of the model, specifically in the case of quartic hilltop and radiatively corrected Higgs inflation. arXiv: http://arxiv.org/abs/http://arxiv.org/abs/2208.09390v2

Regularization of Single Field Inflation Models by Josh Hoffmann et al. on Thursday 24 November There are many single field inflationary models that are consistent with the recent Planck 2018 measurements of the spectral index $n_s$ and tensor-to-scalar ratio $r$. Despite good agreement with observational data some of these models suffer from having unregularized potentials which would produce a collapsing universe shortly after the end of inflation. In this paper we show that how one chooses to correct the behaviour potential towards the end of inflation can have a significant effect on the inflationary predictions of the model, specifically in the case of quartic hilltop and radiatively corrected Higgs inflation. arXiv: http://arxiv.org/abs/http://arxiv.org/abs/2208.09390v2

“If Germany goes recreational,  everybody's going to follow them to some extent. If the leader says, look, this is how I'm addressing the product, the issue that I see at hand, you can pretty much be sure all the bigger countries will come in line like France, Portugal… And a few others are going to really move quickly because there's no reason for them to be disadvantaged if the rules are already set.” - Michael SassanoWelcome back to the To Be Blunt podcast! In this episode, Shayda Torabi welcomes Michael Sassano of SOMAI Pharmaceuticals to take a broader look at the global cannabis market. He brings his wealth of experience to dive deep into European cannabis production, innovation, and legalization and juxtaposes the current landscape with what's happening in North America. He also talks about SOMAI and how they are producing high-quality medical cannabis treatments for patients all over the world. [00:00 - 09:05] Thoughts on My Germany Trip[09:06- 21:10] Europe's Leading Cannabis Markets[21:11 - 37:04] Regularization and Comparing Recreational vs. Medicinal[37:05 - 46:52] The Pharmaceutical Industry and Innovation[46:53 - 53:42] Marketing Cannabis to European Consumers[53:43 - 1:03:41] What It Takes to Be Licensed and to Operate in Europe[1:03:42 - 1:14:40] A Look at the US Market and The Future of SOMAI Michael is the CEO and Chairman of the Board for SOMAI Pharmaceuticals LTD, a European pharmaceutical and biotech company centered on manufacturing in Lisbon, Portugal, and distribution of EU GMP-certified cannabinoid-containing pharmaceuticals throughout the European Union and globally. SOMAI emphasizes scientific pharmacology applications with EU-GMP standards to deliver treatments to the endocannabinoid system, effectively and with consistency across all markets. Taking with him the product development knowledge honed in the competitive American market, SOMAI is the largest and most advanced cannabinoid manufacturing facility across legal European markets producing medicines products and registered API's.Connect with Michael!Check out somaipharma.eu! Watch this episode on YouTube:https://www.youtube.com/watch?v=66yTNFlh20MShayda Torabi has been called one of the most influential Women in WordPress and now she's one of the women leading the cannabis reformation conversation building one of Texas' premier CBD brands. She's currently the CEO and Co-Founder of RESTART CBD, a female-run education first CBD wellness brand. And has formerly held marketing positions at WP Engine and WebDevStudios. Shayda is the host of a podcast for cannabis marketers called To Be Blunt, where she interviews top cannabis brands on their most successful marketing initiatives. When Shayda's not building her cannabiz in Texas, you can find her on the road exploring the best hikes and spots for vegan ice cream. Follow Shayda at @theshaydatorabi SPONSORSHIP is brought to you by Restart CBD. Check them out for your CBD needsLEAVE A REVIEW + help someone who wants to join me for episodes featuring some serious cannabis industry by sharing this episode or click here to listen to past episodesRESTART CBD is an education-first CBD wellness brand shipping nationwide.RESTART CBD RESTART CBD is an education first CBD wellness brand shipping nationwide. restartcbd.com

“I'm glad that we have the innovation that we do, and I think that the process of synthesis is going to lead us to even greater discoveries.” - Tyler RoachWelcome back to the To Be Blunt podcast! In this episode, Shayda Torabi welcomes Tyler Roach, “The HHC Guy” and VP of Sales of Colorado Chromatography, to share a dialogue about synthesizing cannabinoids. Tyler goes in-depth about understanding HHC and the cannabinoids alphabet soup, tailoring medicine through synthesis, and bringing innovation into the industry in the safest way possible.[00:00 - 018:05] Opening the Conversation to the Chemistry of Cannabinoids[13:11 - 22:48] Educating Consumers About the Cannabinoids Alphabet Soup[22:49 - 38:58] Differentiating HHC and Other Cannabinoids[38:59 - 43:48] Legalization and Regularization of Chemically Derived Cannabinoids[43:49 - 50:58] Cannabinoids in a Pharma Perspective [50:59 - 1:21:50] Consumer Safety and Important Considerations on Dosing and Potency Tyler was born in southern California and moved with his family to Colorado summer of 2004. While in high school, Tyler started working for his father's carpentry business. After graduating high school, he continued to work in the family business and grew experience in sales and operations management. All this time, he knew that the carpentry industry was not his forever place. He always had an interest in the hemp and cannabis industry and would expand his knowledge when he could. In Spring of 2021, Tyler had lunch with the executive team of Colorado Chromatography Labs and within a week was offered the job, Director of Sales. He has worked with the team to bring HHC to market and has helped the company grow tremendously. After 9 months with Colorado Chromatography Labs, Tyler was promoted to VP of Sales. Connect with Tyler!Visit the Colorado Chromatography website! Shayda Torabi has been called one of the most influential Women in WordPress and now she's one of the women leading the cannabis reformation conversation building one of Texas' premier CBD brands. She's currently the CEO and Co-Founder of RESTART CBD, a female-run education first CBD wellness brand. And has formerly held marketing positions at WP Engine and WebDevStudios. Shayda is the host of a podcast for cannabis marketers called To Be Blunt, where she interviews top cannabis brands on their most successful marketing initiatives. When Shayda's not building her cannabiz in Texas, you can find her on the road exploring the best hikes and spots for vegan ice cream. Follow Shayda at @theshaydatorabi Key Quote:“There is a big place for synthesis within the industry, but there's a big misunderstanding that a lot of these molecules that you're already consuming on a daily basis are synthesized. There is a process done in a lab to get these molecules, and there's good to be had with that as long as we follow safe procedures and we have proper testing.” - Tyler RoachSPONSORSHIP is brought to you by Restart CBD. Check them out for your CBD needsLEAVE A REVIEW + help someone who wants to join me for episodes featuring some serious cannabis industry by sharing this episode or click here to listen to past episodesRESTART CBDRESTART CBD is an education-first CBD wellness brand shipping nationwide. restartcbd.com

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Regularization Causes Modularity Causes Generalization, published by dkirmani on January 1, 2022 on LessWrong. Epistemic Status: Exploratory Things That Cause Modularity In Neural Networks Modularity is when a neural network can be easily split into several modules: groups of neurons that connect strongly with each other, but have weaker connections to outside neurons. What, empirically, makes a network become modular? Several things: Filan et al.[1]: Training a model with dropout Weight pruning L1/L2 regularization Kashtan & Alon: Switching between one objective function and a different (but related[2]) objective function every 20 generations Clune et al.: Adding penalties for connections between neurons Modularity Improves Generalization What good is modularity? Both Clune et al. and Kashtan & Alon agree: more modular networks are more adaptable. They make much more rapid progress towards their goals than their non-modular counterparts do: Modular neural networks, being more adaptable, make faster progress towards their own goals. Not only that, but their adaptability allows them to rapidly advance on related[2:1] goals as well; if their objective function was to suddenly switch to a related goal, they would adapt to it much quicker than their non-modular counterparts. In fact, modular neural networks are so damn adaptable that they do better on related goals despite never training on them. That's what generalization is: the ability to perform well at tasks with little to no previous exposure to them. That's why we use L1/L2 regularization, dropout, and other similar tricks to make our models generalize from their training data to their validation data. These tricks work because they increase modularity, which, in turn, makes our models better at generalizing to new data. How Dropout Causes Modularity What's true for the group is also true for the individual. It's simple: overspecialize, and you breed in weakness. It's slow death. Major Kusanagi, Ghost in the Shell Training with dropout is when you train a neural network, but every neuron has a chance of 'dropping out': outputting zero, regardless of its input. In practice, making 20-50% of your model's neurons spontaneously fail during training usually makes it much better at generalizing to previously unseen data. Ant colonies have dropout. Ants die all the time; they die to war, to famine, and to kids with magnifying glasses. In response, anthills have a high bus factor. Not only do anthills have specialist ants that are really good at nursing, foraging, and fighting, they also have all-rounder ants that can do any of those jobs in an emergency: Dropout incentivizes robustness to random module failures. One way to be robust to random module failures is to have modules that have different specialties, but can also cover for each other in a pinch. Another way is to have a bunch of modules that all do the exact same thing. For a static objective function, from the perspective of an optimizer: If you expect a really high failure rate (like 95%), you should make a bunch of jack-of-all-trades modules that're basically interchangeable. If you expect a moderate failure rate (like 30%), you should make your modules moderately specialized, but somewhat redundant. Like ants! If you expect no failures at all, you should let modules be as specialized as possible in order to maximize performance. Do that, and your modules end up hyperspecialized and interdependent. The borders between different modules wither away; you no longer have functionally distinct modules to speak of. You have a spaghetti tower. Why would modules blur together? "Typically, there are many possible connections that break modularity and increase fitness. Thus, even an initially modular solution rapidly evolves into one of many possible non-modular soluti...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Regularization Causes Modularity Causes Generalization, published by dkirmani on January 1, 2022 on LessWrong. Epistemic Status: Exploratory Things That Cause Modularity In Neural Networks Modularity is when a neural network can be easily split into several modules: groups of neurons that connect strongly with each other, but have weaker connections to outside neurons. What, empirically, makes a network become modular? Several things: Filan et al.[1]: Training a model with dropout Weight pruning L1/L2 regularization Kashtan & Alon: Switching between one objective function and a different (but related[2]) objective function every 20 generations Clune et al.: Adding penalties for connections between neurons Modularity Improves Generalization What good is modularity? Both Clune et al. and Kashtan & Alon agree: more modular networks are more adaptable. They make much more rapid progress towards their goals than their non-modular counterparts do: Modular neural networks, being more adaptable, make faster progress towards their own goals. Not only that, but their adaptability allows them to rapidly advance on related[2:1] goals as well; if their objective function was to suddenly switch to a related goal, they would adapt to it much quicker than their non-modular counterparts. In fact, modular neural networks are so damn adaptable that they do better on related goals despite never training on them. That's what generalization is: the ability to perform well at tasks with little to no previous exposure to them. That's why we use L1/L2 regularization, dropout, and other similar tricks to make our models generalize from their training data to their validation data. These tricks work because they increase modularity, which, in turn, makes our models better at generalizing to new data. How Dropout Causes Modularity What's true for the group is also true for the individual. It's simple: overspecialize, and you breed in weakness. It's slow death. Major Kusanagi, Ghost in the Shell Training with dropout is when you train a neural network, but every neuron has a chance of 'dropping out': outputting zero, regardless of its input. In practice, making 20-50% of your model's neurons spontaneously fail during training usually makes it much better at generalizing to previously unseen data. Ant colonies have dropout. Ants die all the time; they die to war, to famine, and to kids with magnifying glasses. In response, anthills have a high bus factor. Not only do anthills have specialist ants that are really good at nursing, foraging, and fighting, they also have all-rounder ants that can do any of those jobs in an emergency: Dropout incentivizes robustness to random module failures. One way to be robust to random module failures is to have modules that have different specialties, but can also cover for each other in a pinch. Another way is to have a bunch of modules that all do the exact same thing. For a static objective function, from the perspective of an optimizer: If you expect a really high failure rate (like 95%), you should make a bunch of jack-of-all-trades modules that're basically interchangeable. If you expect a moderate failure rate (like 30%), you should make your modules moderately specialized, but somewhat redundant. Like ants! If you expect no failures at all, you should let modules be as specialized as possible in order to maximize performance. Do that, and your modules end up hyperspecialized and interdependent. The borders between different modules wither away; you no longer have functionally distinct modules to speak of. You have a spaghetti tower. Why would modules blur together? "Typically, there are many possible connections that break modularity and increase fitness. Thus, even an initially modular solution rapidly evolves into one of many possible non-modular soluti...

Today, I will talk about the case of Ram Manohar Joint Hospital and Others v. Munna Prasad Saini and Another, Civil Appeal No. 5810 of 2021, wherein the Hon'ble Supreme Court discussed whether a contractual employee working in a government establishment is entitled for reinstatement on account of his illegal termination or not.To know more about it, please visit https://www.desikanoon.co.in/2021/09/reinstatement-regularization-industrial-disputes.htmlTelegram: https://t.me/Legal_Talks_by_DesiKanoonYouTube Channel: https://www.youtube.com/channel/UCMmVCFV7-Kfo_6S42kPhz2wApple Podcasts: https://podcasts.apple.com/us/podcast/legal-talks-by-desikanoon/id1510617120Spotify: https://open.spotify.com/show/3KdnziPc4I73VfEcFJa59X?si=vYgrOEraQD-NjcoXA2a7Lg&dl_branch=1&nd=1Google Podcasts: https://podcasts.google.com/feed/aHR0cHM6Ly9mZWVkcy5zaW1wbGVjYXN0LmNvbS84ZTZTcGREcw?sa=X&ved=2ahUKEwiuz4ifzpLxAhVklGMGHb4HAdwQ9sEGegQIARADAmazon Music: https://music.amazon.com/podcasts/4b89fb71-1836-414e-86f6-1116324dd7bc/Legal-Talks-by-Desikanoon Please subscribe and follow us on YouTube, Instagram, iTunes, Twitter, LinkedIn, Discord, Telegram and Facebook. Credits: Music by Wataboi from Pixabay Thank you for listening!

On today's show, we will discuss the case of Vice Chancellor Anand Agriculture University v. Kanubhai Nanubhai Vaghela and Another, 2021 SCC OnLine SC 491, wherein the Hon'ble Supreme Court discussed whether the daily wagers are entitled for regularization of their services. To read more about it, please visit our Blog https://www.desikanoon.co.in/2021/07/daily-wagers-wage-workers-supreme-court-regularization.htmlTelegram: https://t.me/Legal_Talks_by_DesiKanoonYouTube Channel: https://www.youtube.com/channel/UCMmVCFV7-Kfo_6S42kPhz2wApple Podcasts: https://podcasts.apple.com/us/podcast/legal-talks-by-desikanoon/id1510617120Spotify: https://open.spotify.com/show/3KdnziPc4I73VfEcFJa59X?si=vYgrOEraQD-NjcoXA2a7Lg&dl_branch=1&nd=1Google Podcasts: https://podcasts.google.com/feed/aHR0cHM6Ly9mZWVkcy5zaW1wbGVjYXN0LmNvbS84ZTZTcGREcw?sa=X&ved=2ahUKEwiuz4ifzpLxAhVklGMGHb4HAdwQ9sEGegQIARADAmazon Music: https://music.amazon.com/podcasts/4b89fb71-1836-414e-86f6-1116324dd7bc/Legal-Talks-by-Desikanoon Please subscribe and follow us on YouTube, Instagram, iTunes, Twitter, LinkedIn, Discord, Telegram and Facebook. Credits: Music by Wataboi from Pixabay Thank you for listening!

Rigel Insights Podcast Show A Profit & Solutions Research Media Productions Rigel Insights Podcast Show is podcast-based created by me to share my thoughts, ideas & takes on marketing, investment, real estate, technology, traveling, global life, and anything that enjoy by me. More Ways to Listen Breaker Podcast: https://www.breaker.audio/rigel-insig... Google Podcast: https://podcasts.google.com/feed/aHR0... Pocket Cast: https://pca.st/b7k01j99 Radio Public: https://radiopublic.com/rigel-insight... Spotify: https://open.spotify.com/show/4W722Lr... RSS Feed: https://anchor.fm/s/4a5f9084/podcast/rss Achor Podcast: https://anchor.fm/rigelinsights About me Deb Bandyopadhyay Digital & Traditional Marketing Consultant | Technology Consultant | Serial Entrepreneur | Investor & Trader Find me anywhere @debadipb Email:debadipb@gmail.com LinkedIn Profile: https://linkedin.com/in/debadipb Website: https://debadip.co PS Website: http://profitsolutions.tk USA | Canada | India | UK | Italy | Rest of World #debadipb #profitsolutions --- This episode is sponsored by · Anchor: The easiest way to make a podcast. https://anchor.fm/app

In the fourth episode of The Venezuela Briefing, hosts Geoff Ramsey and Kristen Martinez-Gugerli interview WOLA colleague Gimena Sanchez and Dejusticia's Lucia Ramirez to learn more about a recent measure by the Colombian government to provide formal status to Venezuelan migrants living in the country. This permit would offer access to work authorization, formal employment, and health care services for as many as 1.7 million Venezuelan migrants for a period of 10 years. Colombia, which hosts the largest number of Venezuelan migrants and refugees out of any other country in Latin America, announced the decree on February 8. While this is the broadest measure to date to address the Venezuelan displacement crisis, there is far more to be done to protect Venezuelan migrants in need.

This newsletter is really a public policy thought-letter. While excellent newsletters on specific themes within public policy already exist, this thought-letter is about frameworks, mental models, and key ideas that will hopefully help you think about any public policy problem in imaginative ways. It seeks to answer just one question: how do I think about a particular public policy problem/solution?PS: If you enjoy listening instead of reading, we have this edition available as an audio narration on all podcasting platforms courtesy the good folks at Ad-Auris. If you have any feedback, please send it to us.- Pranay Kotasthane & RSJIt’s the 72nd Republic Day today (as we write this). The Constitution of India is a revolutionary document. The past few years have seen some wonderful works of scholarship about our constitution. We’d like to call out a few here. The Constitution will remain a distant and daunting document as long as we don’t develop a habit of referencing it directly. Thankfully, EBC has been publishing a coat pocketbook version since 2009. It works great as a reference book. And it doesn’t hurt that it looks elegant and makes for a wonderful gift. As an aside, a dream project of mine is to convert the constitution into a knowledge graph that visually connects the cross-referencing articles in the Constitution. If any AtU reader has the technical chops to make this happen, please ping us. Madhav Khosla’s India’s Founding Moment: The Constitution of a Most Surprising Democracyis a brilliant and erudite work that is essential to understand the radical nature of our Constitution and the interplay of ideas and debates among people who cared deeply for this nation that led to its creation.Rohit De’s A People’s Constitution: The Everyday Life of Law in the Indian Republic challenges the idea that our Constitution was the product of an elitist imagination whose impact in the lives of ordinary Indians was minimal. De uses four examples to make his case about the Constitution empowering the people of India to take on the state. Tripurdaman Singh’s Sixteen Stormy Days: The Story of the First Amendment of Constitution of India shows how the idealism of the Constitution became a difficult burden to bear while running a government. No constitution can be eternally infallible. But we set a precedent of changing the architecture of our Constitution really early in the life of our Republic. We might never recover from that ‘original sin’.Gautam Bhatia’s The Transformative Constitution: A Radical Biography in Nine Actsexplains in detail why the Constitution at its core aims to bring about a social revolution. Many a constitution aim to transform the polity and economy but few aim to change society itself. This is what sets the Indian Constitution apart. The classic work on this line of reasoning is Granville Austin’s The Indian Constitution: Cornerstone of a Nation. This passage from Rohit De’s A People’s Constitution helps make sense of the remarkable achievement that the Indian Constitution is.“The Indian Constitution was written over a period of four years by the Constituent Assembly. Dominated by the Congress Party, India’s leading nationalist political organization, the assembly sought to include a wide range of political opinions and represented diversity by sex, religion, caste, and tribe. This achievement is striking compared to other states that were decolonized. Indians wrote the Indian Constitution, unlike the people of most former British colonies, like Kenya, Malaysia, Ghana, and Sri Lanka, whose constitutions were written by British officials at Whitehall. Indian leaders were also able to agree upon a constitution, unlike Israeli and Pakistani leaders, both of whom elected constituent assemblies at a similar time but were unable to reach agreement on a document. The Indian Constitution is the longest surviving constitution in the post-colonial world, and it continues to dominate public life in India. Despite this, its endurance has received little attention from scholars. [Rohit De, A People’s Constitution, pg 2]To explain the last point visually here’s a comparative chart plotting the number of constitutions against the year of independence for Asian states. Source: Asia’s tryst with constitutionalism, Pranay Kotasthane and Piyush Singh, PragatiIndia, of course, falls in the heavily clustered zone labelled “independence era states” - political communities that overthrew European colonialism to establish new nation-states. Zooming in this era, we find that a handful of constitutions in Asia have survived. Only three amongst them had a constituent assembly that brought together people to make their own constitutions. And even amongst the ones where one constitution has survived this far, most have been beset by politically active militaries and dictatorships. The Indian constitution is without a doubt an exception to be proud of. Source: Asia’s tryst with constitutionalism, Pranay Kotasthane and Piyush Singh, PragatiAmong the many flaws that are often pointed out about our Constitution, the one we disagree most with is about how unmoored it was from India’s past. The accusation is we didn’t try and locate the great ideas and values of the Constitution in our past. This created a sense of distance of the ordinary Indian from the ideals of the people’s Constitution. We have two problems with this line of argument. First, this alienation from our tradition (if true) was to be bridged by later scholars and interpreters of the Constitution including legislators and administrators. The members of the Constituent Assembly shouldn’t have been expected to also write a commentary on it in parallel. Second, there are multiple references to how the principles enshrined in it are consistent with the best of our historical tradition.Another common criticism is that the Constitution reproduced two-thirds of the GoI Act of 1935 and hence wasn’t transformative enough. What’s forgotten of course is that the 1935 Act itself was a result of a powerful Indian independence movement’s consistent political pressure on the British government. The charge of not being transformative enough” needs rethinking.We will leave you with two excerpts of speeches made in the Assembly that support the assertion that the Constitution is consistent with the best of the Indian historical traditions while leaving out the undesirable elements. On 17th October 1949, J.B. Kripalani made both the points we have stated above:“Sir, I want, at this solemn hour to remind the House that what we have stated in this Preamble are not legal and political principles only. They are also great moral and spiritual principles and if I may say so, they are mystic principles. In fact these were not first legal and constitutional principles, but they were really spiritual and moral principles. If we look at history, we shall find that because the lawyers and politician made their principles into legal and constitutional form that their life and vitality was lost and is being lost even today. Take democracy. What is it? It implies the equality of man, it implies fraternity. Above all it implies the great principle of non-violence. How can there be democracy where there is violence? Even the ordinary definition of democracy is that instead of breaking heads, we count heads. This non-violence then there is at the root of democracy. And I submit that the principle of non-violence, is a moral principle. It is a spiritual principle. It is a mystic principle. It is a principle which says that life is one, that you cannot divide it, that it is the same life pulsating through us all. As the Bible puts it, "we are one of another," or as Vendanta puts it, that all this is One. If we want to use democracy as only a legal, constitutional and formal device, I submit, we shall fail. As we have put democracy at the basis of your Constitution, I wish Sir, that the whole country should understand the moral, the spiritual and the mystic implication of the word "democracy". If we have not done that, we shall fail as they have failed in other countries. Democracy will be made into autocracy and it will be made into imperialism, and it will be made into fascism. But as a moral principle, it must be lived in life. If it is not lived in life, and the whole of it in all its departments, it becomes only a formal and a legal principal. We have got to see that we live this democracy in our life.” And more famously, Dr. B.R. Ambedkar in his last speech at the Constituent Assembly draws upon our past: “It is not that India did not know what Democracy is. There was a time when India was studded with republics, and even where there were monarchies, they were either elected or limited. They were never absolute. It is not that India did not know Parliaments or Parliamentary Procedure. A study of the Buddhist Bhikshu Sanghas discloses that not only there were Parliaments-for the Sanghas were nothing but Parliaments – but the Sanghas knew and observed all the rules of Parliamentary Procedure known to modern times. They had rules regarding seating arrangements, rules regarding Motions, Resolutions, Quorum, Whip, Counting of Votes, Voting by Ballot, Censure Motion, Regularization, Res Judicata, etc.Although these rules of Parliamentary Procedure were applied by the Buddha to the meetings of the Sang has, he must have borrowed them from the rules of the Political Assemblies functioning in the country in his time.This democratic system India lost. Will she lose it a second time? I do not know. But-it is quite possible in a country like India – where democracy from its long disuse must be regarded as something quite new – there is danger of democracy giving place to dictatorship. It is quite possible for this newborn democracy to retain its form but give place to dictatorship in fact. If there is a landslide, the danger of the second possibility of becoming actuality is much greater.”The above lines are followed by his famous ‘Grammar of Anarchy’ passage. He knew what he was talking about.HomeWorkReading and listening recommendations on public policy matters[Video] Ambedkar’s speech at the Constituent Assembly [Document] Judgment on Shankari Prasad vs Union of India: The First Constitution Amendment Act, 1951 was challenged in the Shankari Prasad vs. Union of India case. The Supreme Court held that the Parliament, under Article 368, has the power to amend any part of the constitution including fundamental rights. [Podcast] On Puliyabaazi, Rohit De joins Saurabh and Pranay to discuss India’s tryst with constitutionalism.[Article] Democracy and the Republic - the differences between these two concepts. Get on the email list at publicpolicy.substack.com

Undocumented workers and asylum seekers in Canada say the federal government’s new regularization program is deeply unfair to the hundreds of thousands of non-status workers who have being doing essential work during the Covid-19 pandemic. We speak with Mohamed Barry, an organizer with Solidarity Across Borders in Montreal.

Undocumented workers and asylum seekers in Canada say the federal government’s new regularization program is deeply unfair to the hundreds of thousands of non-status workers who have being doing essential work during the Covid-19 pandemic. We speak with Mohamed Barry, an organizer with Solidarity Across Borders in Montreal.

आज लखनऊ स्मार्ट न्यूज़ में सुनिए, लखनऊ में 300 बेड का कोरोना अस्पताल प्रस्तावित किया गया है KGMU ने राज्य के लिए पहला प्लाज्मा बैंक शुरू किया | लखनऊ विकास प्राधिकरण अनियमित कॉलोनियों के नियमितीकरण से शुरू होता है |

Link to bioRxiv paper: http://biorxiv.org/cgi/content/short/2020.08.14.251835v1?rss=1 Authors: Cao, C., Greenberg, M., Long, Q. Abstract: Many tools can reconstruct viral sequences based on next generation sequencing reads. Although existing tools effectively recover local regions, their accuracy suffers when reconstructing the whole viral genomes (strains). Moreover, they consume significant memory when the sequencing coverage is high or when the genome size is large. We present WgLink to meet this challenge. WgLink takes local reconstructions produced by other tools as input and patches the resulting segments together into coherent whole-genome strains. We accomplish this using an L_0+L_1-regularized regression synthesizing variant allele frequency data with physical linkage between multiple variants spanning multiple regions simultaneously. WgLink achieves higher accuracy than existing tools both on simulated and real data sets while using significantly less memory (RAM) and fewer CPU hours. Source code and binaries are freely available at https://github.com/theLongLab/wglink. Copy rights belong to original authors. Visit the link for more info

Link to bioRxiv paper: http://biorxiv.org/cgi/content/short/2020.07.29.227546v1?rss=1 Authors: Zhang, R., Atwal, G. S., Lim, W. Abstract: With the rapid advancement of single-cell RNA-seq (scRNA-seq) technology, many data preprocessing methods have been proposed to address numerous systematic errors and technical variabilities inherent in this technology. While these methods have been demonstrated to be effective in recovering individual gene expression, the suitability to the inference of gene-gene associations and subsequent gene networks reconstruction have not been systemically investigated. In this study, we benchmarked five representative scRNA-seq normalization/imputation methods on human cell atlas bone marrow data with respect to their impact on inferred gene-gene associations. Our results suggested that a considerable amount of spurious correlations was introduced during the data preprocessing steps due to over-smoothing of the raw data. We proposed a model-agnostic noise regularization method that can effectively eliminate the correlation artifacts. The noise regularized gene-gene correlations were further used to reconstruct gene co-expression network and successfully revealed several known immune cell modules. Copy rights belong to original authors. Visit the link for more info

Visualizing your network to understand why it work

In this episode with talk about regularization, an effective technique to deal with overfitting by reducing the variance of the model. Two techniques are introduced: ridge regression and lasso. The latter one is effectively a feature selection algorithm.

If you have a probationary employee who demands that they be regularized, are you obligated to do that? How do you decide whether to let them continue in your team or let them go? What measuring stick or benchmark would be useful in deciding?

第４３回では，Byte Pair Encodingを用いたサブワード正則化手法，BPE-dropoutを解説しました． 今回紹介した記事はこちらのissueで解説しています． https://github.com/jojonki/arXivNotes/issues/302 サポーターも募集中です． https://www.patreon.com/jojonki --- Support this podcast: https://anchor.fm/lnlp-ninja/support

Learn about the impact of DOLE's regularization program on Small Business. | Atty. Jericho "Jake" Del Puerto | www.laborlaw.ph | info@laborlaw.ph

ニューラル自然言語処理の前処理として複雑な単語を限られた語彙集合で分割するアルゴリズムについて向井が話します。

listen to an interview with interdisciplinary artist Amanda Ruiz speaking about various exhibitions and art installations in both Mexico City and Montreal. Amanda also speaks about collaborations with the activist organization Mexicans United for Regularization which has been fighting the deportation of Mexican asylum seeker and for the regularization of non-status Mexican people. this interview was recorded / produced for broadcast on CKUT fm by Stefan @spirodon Christoff.

Lots of data is usually seen as a good thing. And it is a good thing--except when it's not. In a lot of fields, a problem arises when you have many, many features, especially if there's a somewhat smaller number of cases to learn from; supervised machine learning algorithms break, or learn spurious or un-interpretable patterns. What to do? Regularization can be one of your best friends here--it's a method that penalizes overly complex models, which keeps the dimensionality of your model under control.

Thu, 26 Nov 2015 12:00:00 +0100 https://edoc.ub.uni-muenchen.de/19007/ https://edoc.ub.uni-muenchen.de/19007/1/Schauberger_Gunther.pdf Schauberger, Gunther ddc:004, ddc:000, Fakultät für Mathematik, Informatik und Statisti

Deep learning machine technology has rapidly developed over the past five years due in part to a variety of actors such as: better technology, convolutional net algorithms, rectified linear units, and a relatively new learning strategy called "dropout" in which hidden unit feature detectors are temporarily deleted during the learning process. This article introduces and discusses the concept of "dropout" to support deep learning performance and makes connections of the "dropout" concept to concepts of regularization and model averaging. For more details and background references, check out: www.learningmachines101.com !  

Sliced Inverse Regression (SIR) is an effective method for dimension reduction in highdimensional regression problems. The original method, however, requires the inversion of the predictors covariance matrix. In case of collinearity between these predictors or small sample sizes compared to the dimension, the inversion is not possible and a regularization technique has to be used. Our approach is based on an interpretation of SIR axes as solutions of an inverse regression problem. A prior distribution is then introduced on the unknown parameters of the inverse regression problem in order to regularize their estimation. We show that some existing SIR regularizations can enter our framework, which permits a global understanding of these methods. Three new priors are proposed, leading to new regularizations of the SIR method, and compared on simulated data. An application to the estimation of Mars surface physical properties from hyperspectral images is provided.

The majority of regularization methods in regression analysis has been designed for metric predictors and can not be used for categorical predictors. A rare exception is the group lasso which allows for categorical predictors or factors. We will consider alternative approaches based on penalized likelihood and boosting techniques. Typically the operating model will be a generalized linear model. We will start with ordered categorical predictors which unfortunately are often treated as metric variables because software is available. It is shown how difference penalties on adjacent dummy coefficients can be used to obtain smooth effect curves that can be estimated also in cases where simple maximum likelihood methods fail. The difference penalty turns out to be highly competitive when compared to methods often seen in practice, namely simple linear regression on the group labels and pure dummy coding. In a second step L1-penalty based methods that enforce variable selection and clustering of categories are presented and investigated. It is distinguished between ordered predictors where clustering refers to the fusion of adjacent categories and nominal predictors for which arbitrary categories can be fused. The methods allow to identify which categories do actually differ with respect to the dependent variable. Finally interaction effects are modeled within the framework of varying coefficients models. For the proposed methods properties of the estimators are investigated. Methods are illustrated and compared in simulation studies and applied to real world data.

The scientific investigation of the solid Earth's complex processes, including their interactions with the oceans and the atmosphere, is an interdisciplinary field in which seismology has one key role. Major contributions of modern seismology are (1) the development of high-resolution tomographic images of the Earth's structure and (2) the investigation of earthquake source processes. In both disciplines the challenge lies in solving a seismic inverse problem, i.e. in obtaining information about physical parameters that are not directly observable. Seismic inverse studies usually aim to find realistic models through the minimization of the misfit between observed and theoretically computed (synthetic) ground motions. In general, this approach depends on the numerical simulation of seismic waves propagating in a specified Earth model (forward problem) and the acquisition of illuminating data. While the former is routinely solved using spectral-element methods, many seismic inverse problems still suffer from the lack of information typically leading to ill-posed inverse problems with multiple solutions and trade-offs between the model parameters. Non-linearity in forward modeling and the non-convexity of misfit functions aggravate the inversion for structure and source. This situation requires an efficient exploitation of the available data. However, a careful analysis of whether individual models can be considered a reasonable approximation of the true solution (deterministic approach) or if single models should be replaced with statistical distributions of model parameters (probabilistic or Bayesian approach) is inevitable. Deterministic inversion attempts to find the model that provides the best explanation of the data, typically using iterative optimization techniques. To prevent the inversion process from being trapped in a meaningless local minimum an accurate initial low frequency model is indispensable. Regularization, e.g. in terms of smoothing or damping, is necessary to avoid artifacts from the mapping of high frequency information. However, regularization increases parameter trade-offs and is subjective to some degree, which means that resolution estimates tend to be biased. Probabilistic (or Bayesian) inversions overcome the drawbacks of the deterministic approach by using a global model search that provides unbiased measures of resolution and trade-offs. Critical aspects are computational costs, the appropriate incorporation of prior knowledge and the difficulties in interpreting and processing the results. This work studies both the deterministic and the probabilistic approach. Recent observations of rotational ground motions, that complement translational ground motion measurements from conventional seismometers, motivated the research. It is investigated if alternative seismic observables, including rotations and dynamic strain, have the potential to reduce non-uniqueness and parameter trade-offs in seismic inverse problems. In the framework of deterministic full waveform inversion a novel approach to seismic tomography is applied for the first time to (synthetic) collocated measurements of translations, rotations and strain. The concept is based on the definition of new observables combining translation and rotation, and translation and strain measurements, respectively. Studying the corresponding sensitivity kernels assesses the capability of the new observables to constrain various aspects of a three-dimensional Earth structure. These observables are generally sensitive only to small-scale near-receiver structures. It follows, for example, that knowledge of deeper Earth structure are not required in tomographic inversions for local structure based on the new observables. Also in the context of deterministic full waveform inversion a new method for the design of seismic observables with focused sensitivity to a target model parameter class, e.g. density structure, is developed. This is achieved through the optimal linear combination of fundamental observables that can be any scalar measurement extracted from seismic recordings. A series of examples illustrate that the resulting optimal observables are able to minimize inter-parameter trade-offs that result from regularization in ill-posed multi-parameter inverse problems. The inclusion of alternative and the design of optimal observables in seismic tomography also affect more general objectives in geoscience. The investigation of the history and the dynamics of tectonic plate motion benefits, for example, from the detailed knowledge of small-scale heterogeneities in the crust and the upper mantle. Optimal observables focusing on density help to independently constrain the Earth's temperature and composition and provide information on convective flow. Moreover, the presented work analyzes for the first time if the inclusion of rotational ground motion measurements enables a more detailed description of earthquake source processes. The complexities of earthquake rupture suggest a probabilistic (or Bayesian) inversion approach. The results of the synthetic study indicate that the incorporation of rotational ground motion recordings can significantly reduce the non-uniqueness in finite source inversions, provided that measurement uncertainties are similar to or below the uncertainties of translational velocity recordings. If this condition is met, the joint processing of rotational and translational ground motion provides more detailed information about earthquake dynamics, including rheological fault properties and friction law parameters. Both are critical e.g. for the reliable assessment of seismic hazards.

Wed, 9 Jul 2014 12:00:00 +0100 https://edoc.ub.uni-muenchen.de/17538/ https://edoc.ub.uni-muenchen.de/17538/1/Moest_Stephanie.pdf Möst, Stephanie ddc:510, ddc:500, Fakultät für Mathematik, Informatik und Statistik

Pricop-Jeckstadt, M (University of Bonn) Wednesday 26 March 2014, 14:30-15:00

Burenkov, V (Cardiff University) Friday 14 February 2014, 09:45-10:30

Somersalo, E (Case Western Reserve University) Tuesday 11 February 2014, 11:00-11:45

Aykroyd, R (University of Leeds) Friday 07 February 2014, 13:45-14:30

Betcke, M (University College London) Friday 07 February 2014, 11:45-12:15

This thesis is concerned with the development of flexible continuous-time survival models based on the accelerated failure time (AFT) model for the survival time and the Cox relative risk (CRR) model for the hazard rate. The flexibility concerns on the one hand the extension of the predictor to take into account simultaneously for a variety of different forms of covariate effects. On the other hand, the often too restrictive parametric assumptions about the survival distribution are replaced by semiparametric approaches that allow very flexible shapes of survival distribution. We use the Bayesian methodology for inference. The arising problems, like e. g. the penalization of high-dimensional linear covariate effects, the smoothing of nonlinear effects as well as the smoothing of the baseline survival distribution, are solved with the application of regularization priors tailored for the respective demand. The considered expansion of the two survival model classes enables to deal with various challenges arising in practical analysis of survival data. For example the models can deal with high-dimensional feature spaces (e. g. gene expression data), they facilitate feature selection from the whole set or a subset of the available covariates and enable the simultaneous modeling of any type of nonlinear covariate effects for covariates that should always be included in the model. The option of the nonlinear modeling of covariate effects as well as the semiparametric modeling of the survival time distribution enables furthermore also a visual inspection of the linearity assumptions about the covariate effects or accordingly parametric assumptions about the survival time distribution. In this thesis it is shown, how the p>n paradigm, feature relevance, semiparametric inference for functional effect forms and the semiparametric inference for the survival distribution can be treated within a unified Bayesian framework. Due the option to control the amount of regularization of the considered priors for the linear regression coefficients, there is no need to distinguish conceptionally between the cases pn. To accomplish the desired regularization, the regression coefficients are associated with shrinkage, selection or smoothing priors. Since the utilized regularization priors all facilitate a hierarchical representation, the resulting modular prior structure, in combination with adequate independence assumptions for the prior parameters, enables to establish a unified framework and the possibility to construct efficient MCMC sampling schemes for joint shrinkage, selection and smoothing in flexible classes of survival models. The Bayesian formulation enables therefore the simultaneous estimation of all parameters involved in the models as well as prediction and uncertainty statements about model specification. The presented methods are inspired from the flexible and general approach for structured additive regression (STAR) for responses from an exponential family and CRR-type survival models. Such systematic and flexible extensions are in general not available for AFT models. An aim of this work is to extend the class of AFT models in order to provide such a rich class of models as resulting from the STAR approach, where the main focus relies on the shrinkage of linear effects, the selection of covariates with linear effects together with the smoothing of nonlinear effects of continuous covariates as representative of a nonlinear modeling. Combined are in particular the Bayesian lasso, the Bayesian ridge and the Bayesian NMIG (a kind of spike-and-slab prior) approach to regularize the linear effects and the P-spline approach to regularize the smoothness of the nonlinear effects and the baseline survival time distribution. To model a flexible error distribution for the AFT model, the parametric assumption for the baseline error distribution is replaced by the assumption of a finite Gaussian mixture distribution. For the special case of specifying one basis mixture component the estimation problem essentially boils down to estimation of log-normal AFT model with STAR predictor. In addition, the existing class of CRR survival models with STAR predictor, where also baseline hazard rate is approximated by a P-spline, is expanded to enable the regularization of the linear effects with the mentioned priors, which broadens further the area of application of this rich class of CRR models. Finally, the combined shrinkage, selection and smoothing approach is also introduced to the semiparametric version of the CRR model, where the baseline hazard is unspecified and inference is based on the partial likelihood. Besides the extension of the two survival model classes the different regularization properties of the considered shrinkage and selection priors are examined. The developed methods and algorithms are implemented in the public available software BayesX and in R-functions and the performance of the methods and algorithms is extensively tested by simulation studies and illustrated through three real world data sets.

If you experience any technical difficulties with this video or would like to make an accessibility-related request, please send a message to digicomm@uchicago.edu. Partha Niyogi Memorial Conference: "Learning Vector-valued Functions and Data-dependent Kernels for Manifold Regularization". This conference is in honor of Partha Niyogi, the Louis Block Professor in Computer Science and Statistics at the University of Chicago. Partha lost his battle with cancer in October of 2010, at the age of 43. Partha made fundamental contributions to a variety of fields including language evolution, statistical inference, and speech recognition. The underlying themes of learning from observations and a rigorous basis for algorithms and models permeated his work.

If you experience any technical difficulties with this video or would like to make an accessibility-related request, please send a message to digicomm@uchicago.edu. Partha Niyogi Memorial Conference: "Learning Vector-valued Functions and Data-dependent Kernels for Manifold Regularization". This conference is in honor of Partha Niyogi, the Louis Block Professor in Computer Science and Statistics at the University of Chicago. Partha lost his battle with cancer in October of 2010, at the age of 43. Partha made fundamental contributions to a variety of fields including language evolution, statistical inference, and speech recognition. The underlying themes of learning from observations and a rigorous basis for algorithms and models permeated his work.

Mon, 28 Nov 2011 12:00:00 +0100 https://edoc.ub.uni-muenchen.de/14398/ https://edoc.ub.uni-muenchen.de/14398/1/Petry_Sebastian.pdf Petry, Sebastian ddc:500, ddc:510, Fakultät für Mathematik, Informatik un

Thu, 17 Mar 2011 12:00:00 +0100 https://edoc.ub.uni-muenchen.de/13028/ https://edoc.ub.uni-muenchen.de/13028/1/Scheipl_Fabian.pdf Scheipl, Fabian ddc:000, ddc:004, Fakultät für Mathematik, Informatik und Sta

The case of continuous effect modifiers in varying-coefficient models has been well investigated. Categorial effect modifiers, however, have been largely neglected. In this paper a regularization technique is proposed that allows for selection of covariates and fusion of categories of categorial effect modifiers in a linear model. It is distinguished between nominal and ordinal variables, since for the latter more economic parametrizations are warranted. The proposed methods are illustrated and investigated in simulation studies and real world data evaluations. Moreover, some asymptotic properties are derived.

The case of continuous effect modifiers in varying-coefficient models has been well investigated. Categorial effect modifiers, however, have been largely neglected. In this paper a regularization technique is proposed that allows for selection of covariates and fusion of categories of categorial effect modifiers in a linear model. It is distinguished between nominal and ordinal variables, since for the latter more economic parametrizations are warranted. The proposed methods are illustrated and investigated in simulation studies and real world data evaluations. Moreover, some asymptotic properties are derived. The paper is a preprint of an article that has been accepted for publication in Statistica Sinica. Please use the journal version for citation.

Parametric multicategorical models are an established tool in statistical data analysis. Alternative semi-parametric models are introduced where part of the explanatory variables is still linearly parametrized and the rest is given as a sum of unspecified functions of the explanatory variables. The modelling approach distinguishes between global and category specific variables, in contrast to global variables the latter may have different values for differing categories of the response. Estimation procedures are derived which make use of an expansion in basis functions which are localized on a grid of values of the explanatory variables. Regularization of the estimates is obtained by penalization.