Posts
Comments
Did you use ghost gradients? (gradients that tend to reactivate features that are at zero)
You should train both a feedforward network and a CNN on image classification on imagenet, to see if we see that the hessian of the CNN is more similar to the identity after training rather than the feedforward because of the image understanding priors.
Given that this method returns a numeric matrix, then it must be an Hessian evaluated at a point or the average Hessian of many points. Is the result the Hessian averaged over all training data? Is this average useful rather than just cancelling out high and low Hessian values
You can use the "mp-net2" model from sentence transformers for zero-shot classification (scalar product between the text and the embeddings of "sex" and "violence") decide a cut-off and you are done.
Thanks for the quick response, have you tried fine-tuning the new llama2 models on the data gathered so far to see if there is any interesting results? QLORA is pretty efficient for this.
It looks pretty cool! Adding a Google sign-in option would greatly broaden the reach of the game as most non-technical people do not have a Github account.
We could not find a "speak in French" vector after about an hour of effort, but it's possible we missed something straightforward
Did you try 10 or 20 simple French phrases with a positive sign and their translations with a negative sign?
Also try 1000 english words and 1000 french translations in case scale is the problem.
Also try:
"The following text is in English: ' "
"The following text is in French: ' "
with the second phrase written itself in French.
Very interesting, could you elaborate or give some links?
In my opinion Wearable health is highly neglected because older people are less tech savy than young people, so they use it less than younger people, but they would also benefit much more from the technology. If a 20 year old wears a smart watch that measures and records heart-rate it is almost only for fun, if a 60 year old does it, it could prevent and inform about important issues, but the 20 year old is much more likely to actually use it than the 60 year old.
I also asked ChatGPT, here are the six best ideas that it had (excluding electric bikes, as it was already my idea ;P) (cherry picked by me over 21):
Online education: Online education platforms like Coursera and Khan Academy were mature, widely available, intuitive, and cost no money for basic usage. They also had no regulatory barriers or moral issues and could be used by mutual agreement among one or a few people. Online education also saved a lot of time and played relatively well with the existing format of learning and education.
Digital wallets: Digital wallet technologies like Apple Pay and Google Pay were mature, widely available, intuitive, and cost no money for basic usage. They also had no regulatory barriers or moral issues and could be used by mutual agreement among one or a few people. Digital wallets also saved a lot of time and played relatively well with the existing format of financial transactions and management.
Wearable health monitoring devices: These devices, such as Fitbit and Apple Watch, are mature, widely available, intuitive, and cost no money for basic usage. They also have no regulatory barriers or moral issues and can be used by mutual agreement among one or a few people. They save time and money by allowing users to track their health metrics and stay motivated to maintain a healthy lifestyle.
Online language learning platforms: Platforms like Duolingo and Rosetta Stone are mature, widely available, intuitive, and cost no money for basic usage. They have no regulatory barriers or moral issues and can be used by mutual agreement among one or a few people. They save time and money by providing an accessible and effective way to learn a new language.
Home energy management systems: Devices like Nest Thermostat and Sense Home Energy Monitor are mature, widely available, intuitive, and cost no money for basic usage. They have no regulatory barriers or moral issues and can be used by mutual agreement among one or a few people. They save money and energy by providing users with insights into their home energy usage and allowing them to make informed decisions about energy conservation.
Collaborative writing and editing tools: Platforms like Google Docs and Microsoft Teams are mature, widely available, intuitive, and cost no money for basic usage. They have no regulatory barriers or moral issues and can be used by mutual agreement among one or a few people. They save time and make collaboration more efficient and effective, whether for school projects, business proposals, or creative writing.
Both of them are very reasonable, online education is accessible, almost free, and makes it possible to study even while holding a full time job, from a quick glance a great deal of your requirements are satisfied.
Digital wallets I am less sure about, I never used one, but they look really convenient and easy to use, but I would need more info on how secure they are before starting to use them.
Overall, I think all of these ideas kind of fit your point.
Electric bikes are vastly under-utilized even in European cities where they are safe and effective to use:
- Mature: bike more than 100 years old, electric motors and batteries also mature.
- Cost no money: saves a ton on money over a car
- Was widely available and fairly intuitive for the average person to use: everyone can bike
- Had no regulatory barriers or moral issues: clearly not illegal nor immoral to ride a bike.
- Saved a lot of money and time: saves also time because there is no need for separate exercise.
- Had an immediate payoff: you gain from day 1
- Played relatively well with the existing format of
meetingstransportation - Appealing largely to people in the developed world with high access to information about useful new products and services: high traffic congestion is common there
- Potential daily users in the hundreds of millions: even more, around a third of all persons are potential daily users
The only barriers are perceived risk (not clear if the risk of an accident is higher than the benefit from physical exercise in my opinion, it could well be net positive depending on where you live) and that you look "childish" and kind of weird if you bike to work.
I tried both and neither works
Here is my playthrough with my though process:
:::spoiler
>!0) [2, 4, 6] is VALID
>!Now I think, let's check if the rule is *2
>!1) [31, 62, 93] is VALID
>!Let's check if the rule is always true with 3 random numbers.
>!2) [6534525, 142536, 456342532] is NOT VALID
>!I wanted to check multiply by 3, but I repeated multiply by 2
>!3) [5, 10, 15] is VALID
>!Checking multiply by 3
>!4) [7, 21, 63] is VALID
>!Checking multiply by 10
>!5) [50, 500, 5000] is VALID
>!Now I am thinking: maybe any multiplication is ok? I cannot try them all, let's try constant addition:
>!6) [34, 35, 36] is VALID
>!Let's try constant subtraction
>!7) [5, 4, 3] is NOT VALID
>!Let's try increasing
>!8) [5, 64, 234] is VALID
>!Let's try increasing with random numbers
>!9) [12345, 123456, 1234567] is VALID
>!Let's try increasing or constant.
>!10) [5, 5, 6] is NOT VALID
>!Now I have made up my mind!
>!After 10 tests, your guess was . . .
>!"The sequence is valid if it is strictly increasing."
:::
- CCS does not find the single linear probe with high accuracy: there are more than 20 orthogonal linear probes (i.e. using completely different information) that have similar accuracies as the linear probe found by CCS (for most datasets);
So what about an ensamble of the top 20 linear probes? Is it substantially better than using just the best one alone? I would expect so given that they are orthogonal, so they are using ~uncorrelated information.
The most important thing is approaching other points of view with an open mind, with epistemic humility , that is, knowing that something of what you think can be wrong, even if, from the inside, everything feels right.
On the object level:
- Carbohydrates can be: fruit/whole grains/normal bread/normal pasta or North American crazy industrial snacks and "sugar-cereals" and "sugar-bread". The first one is good, the second one is bad.
- No idea about optimal salt levels, just a note on your language: "potentially deadly" is too vague to have a useful discussion. Crossing the road is "potentially deadly" for some definitions of "potentially deadly". Try to quantify the risk instead, at least approximately.
- For facemasks, a surgical mask that you re-use for several day, without cutting your beard (beard makes it impossible to have a perfect fit), with a random size, removing it constantly to eat and drink, of course it is going to work badly. If you wear a serious P100 masks, of appropriate shape and size, with no beard (cut down to zero with a high quality manual razor) and never remove it, of course it is going to work, it is basic physics at that point. If it is a good idea to wear it, considering costs and benefits, is another discussion.
- Please be really careful about psychoactive substances, they might have long term effects that are little known because they are little studied. (This last statement is based on random stuff read on the internet, take with a grain of salt).
"""
That's when I discovered more effective ways to approach reading, including what I'll call "Guess-and-Check," the technique of scanning and making predictions. Instead of trying to read every word in a textbook, in Guess-and-Check you scan the material and make predictions about what you think the text is saying. This active reading process can help you better engage with the material and activate your prior knowledge. After making your prediction, be sure to confirm or correct it by checking it against the text.
"""
This is similar to the way GPT-3 was trained! Pretty cool that you also found it effective!
Yes, the tone of my comment could be improved. I appreciate him for publishing his lessons to the community and wanted to give some suggestions to improve (eventual) future ones, if he feels like the higher quality is worth the higher effort, and with no obligation. "Al caval donato non si guarda in bocca" (You should not look at the teeth of a gift horse (to learn about its age))
Some suggestions:
- Use a better marker, what you wrote on the whiteboard is almost unreadable for me.
- Expanding on the previous point, write bigger and make better use of the space on the board.
- If you have complex graphics, pre-make them accurately and print them out, put them on the whiteboard with weak tape. (What is the weird of bridge at the start?)
- Use Whisper to make subtitles to help non-native speakers (as another commenter suggested).
- Invest in a tripod to have the camera at a natural height instead of bottom to top.
- I did not watch the lectures, this feedback is from skimming around in the first one.
- In lecture 3, visualizing the Fourier matrix (Discrete Cosine Transform) matrix, could be really interesting.
- Are the lectures ordered by something or following a thread? If so you should make it clear by posting an overview lecture at the start.
- Do not wear glasses, it makes you look weird and lose connection to the viewer.
- Thanks a lot for your effort, we really need to spread mathematical education for the benefit of the whole society.
In Italy we have this, in the Ferragosto week (around the 15 of August) a huge percentage of people is on vacation. In general a lot of people take vacations in August and schools of all order and levels (including university) are closed the whole month.
In the simplest possible way to partecipate, yes, but a hackathon is made to elicit imaginative and novel ways to approach the problem (how? I do not know, it is the partecipants' job to find out).
Yes of course:
Models:
Datasets:
- Laion 400 millions or other sizes: https://laion.ai/blog/laion-400-open-dataset/
- https://paperswithcode.com/dataset/coco-captions
- Imagenet/any image classification dataset: just treat the labels as text, this should be used sparingly as otherwise the model will learn to just output single words.
Also in the performance metric, the sum of the performance of each layer should probably be weighted to give less importance to the initial layers, otherwise we encourage the models to do as much of the work as possible at the start instead of being gradual.
Probably even if not completely by hand, MNIST is so simple that hybrid human-machine optimization could be possible, maybe with a UI where you can see the effect on validation loss in (almost) real time of changing a particular weight with a slider. I do not know if it would be possible to improve the final score by changing the weights one by one. Or maybe the human can use instinctual vision knowledge to improve the convolutional filters.
On Cifar this looks very hard to do manually given that the dataset is much harder than Mnist.
I think that a too large choice of lambda is better than a too small one because if lambda is too big the results will still be interesting (which model architecture is the best under extremely strong regularization?) while if it is too small you will just get a normal architecture slightly more regularized.
The performance can be a weighted average of the final performance and how uniformly we go from totally random to correct. For example if we have 10 refinement models the optimal score in this category can be had when each refinement block reduces the distance from the initial text encoding random vector to the final one by 10% of the original distance each time. This should make sure that the process is in fact gradual, and not that for example, the last two layers do all the work and everything before is just the identity. Also maybe it should not be a linear scale but a logarithmic scale because the final refinements might be harder to do than the initial ones.
Cool GitHub repository, thanks for the link.
Image to text model with successive refinements:
For example, given the image above, the "first" layer of the network outputs: "city", the second one outputs "city with vegetation", the third one "view of a city with vegetation from a balcony", the fourth one "view of a city with skyscrapers on the background and with vegetation from a balcony".
This could be done by starting with a blank description and repeating many times a "detailer" network that should add details to a description given an image.
This should help interpret-ability and thus safety because you are dividing the very complex task of image description into an iteration of a simpler task, of improving an already existing description. Also it would allow you to see exactly where the model went wrong in the case of wrong outputs. Also it might be possible to share weights between the layers to further simplify the model.
A very simple task, like MNIST or CIFAR classification, but the final score is:
where "" is a normalization factor that is chosen to make the tradeoff as interesting as possible. This should be correlated to AI safety as a small and/or very sparse model is much more interpretable and thus safer than a large/dense one. You can work on this for a very long, time, trying simple fully connected neural nets, CNNs, resnets, transformers, autoencoders of any kind and so on. If the task looks too easy you might change it imagenet classification or image generation or anything else, depending on the skill level of the contestants.
Even lower tech: buy very warm and comfy sweaters to wear at home and while sleeping, it could save you a ton of money considering the astronomical power bills that might arrive this winter. Cashmere is really good but expensive.
Extremely cool evolution experiment where E. coli bacteria evolve to eat citrate along with many other interesting happenings.
Yes, I meant plummeting "within reason" (like x10) not plummeting to extremely low values that, as you correctly said, are not possible given the energy cost.
I am not really sure about that. There is not only a huge money cost but also a huge energy cost when sending something into orbit, would the panels even make back the fuel spent to send them? Even if the rocket hardware is reused 100% with no serious maintenance costs (reusing costs more fuel) would the panel even make back that fuel energy alone? I did not do the math but maybe not even that. If we could put them in orbit with a space elevator almost for free the tune would be way different though.
- Very good point: I think the website I linked to refers to peak power, so the Kilowatthours would be lower. (not sure on this, sorry)
- If the panels on orbit last double the time and produce double the energy that is only a factor of 4, while the system is about 300 times more expensive. (but again you have transmission losses that I did not consider)
SpaceX's Falcon 9 now advertises a cost of $62 million to launch 22,800 kg to LEO, $2,720/kg. https://ttu-ir.tdl.org/bitstream/handle/2346/74082/ICES_2018_81.pdf
Given an average solar silicon price of around $9 US per kilogram in 2020 https://www.solarquotes.com.au/blog/solar-silicon-price-hike/#:~:text=Compared%20to%20the%20average%20solar,%2434%20Australian%20dollars%20per%20panel.
This would increase costs 2720 / 9 = 302 times.
The cost of a solar electric system is measured in dollars per watt. The average cost for a residential system is currently $3-5 per watt. That means the average 5-kW residential system will cost $15,000-$25,000, prior to tax credits or incentives. https://sites.energycenter.org/solar/homeowners/cost#:~:text=The%20cost%20of%20a%20solar,to%20tax%20credits%20or%20incentives.
So this system would cost about 4*302 = 1208$ per watt.
This calculation is extremely approximate, but no, it will never work, even if the cost of sending a kg to orbit plummets.
Thanks, added the comment in the correct place now.
Let's have fun with recursion!
A checkerboard where each square is itself a checkerboard.
A cube with mirrors on both sides, the mirrors show multiple reflections of the cube.
A person wearing a shirt with an image of that person wearing that shirt.
Let's have fun with recursion!
A checkerboard where each square is itself a checkerboard.
A cube with mirrors on both sides, the mirrors show multiple reflections of the cube.
A person wearing a shirt with an image of that person wearing that shirt.
Immune erosion is used to make people understand that immune escape is only partial and not total
Thanks for your feedback, in fact correlation is not causation and we must be very careful about self-selection effects. This is not a self-selection effect but still a correlation/causation enigma that I found interesting in recent times: high vitamin D levels were found to be heavily anticorrelated with severe COVID in observational studies, but people of old age are both sensible to severe COVID and have lower vitamin D than average, not only that but people with a healthy lifestyle of many outdoor walks also have higher vitamin D! Is this causation, correlation or both? There is a very interesting article about this on Astral Codex 10 (COVID/Vitamin D, Much More Than You Wanted To Know)
Hi, I wrote the Book-Review on Spark as my first post on LessWrong. Sadly I received no comments in response to it and I would love some feedback after spending so much time writing it. I am open to any kind of feedback about it. I really enjoyed this bounty program and I will probably partecipate also in the future ones.
It might discourage exploration and lead to more stasis in local optimums.
Half as long right?
Given the extreme infectiousness of the Delta strain it is reasonable to assume that everyone will come into contact with it (either with or without illness if the vaccine works properly). If so, if you are already 14+ days from double vaccination, the timing of coming into contact with it now or in three months time is almost irrelevant (if there is space in the hospitals, otherwise it is in fact reasonable to exercise outside a month or two).
Points 3 and 4 can only sound excessive
Where is point 4?
Yes of course
Perception of temperature is mostly cultural yes, what do you think about putting sound absorbing panels on the walls of the room?
Fascinating article, two things:
- On average people prefer a room temperature of 18-22 degrees Celcius.
Are you sure? That looks really cold. My thermodynamics book said 26 C is the best temperature for confort for humans.
2. What about install sound absorbing panels on the walls for sound insulation? Sounds more confortable then earplugs and work both ways, so you can be noisy without giving trouble to the neightbours
Is this comment AI generated?
Incredibly fascinating, I am opposite, only internal verbal monologue, no images at all. I can build step by step an image from simple lines and circles in my mind, but it is a conscious effort, (i.e. I am not really seeing it, if you get what I mean, as soon as I focus on a detail the rest vanishes). Basing on your anecdotal evidence I could maybe learn to imagine images that I am not really seeing with my eyes in that moment but it feels like the opposite of what my mind "is built" to do.
I am quite young, I am in fact in university studying artificial intelligence and this website did play a bit of a role in me choosing this field of study (I was already quite drawn to it before). I think this site has a too extremist view of AI risk, but it is important to read opinions different from mine. This site is mostly quite interesting if not at the level astral codex 10.
I have no concrete advice for you but Good Luck! I wish you all the best.