Modeling sleep patterns

Is there some sort of pattern detecting thing, whose name perhaps includes something like "markov" or "kolmogorov" or "bayesian", that could automatically take a time series data and predict the next values based on an unknown, complex model?

Prediction isn't magic -- you need relevant information. In this case, if information about the causes of your sleep patterns are not in the time series, you won't be able to accomplish much with this approach.

I note that you include info about caffeine and other drug usage in your data. These are obvious possible causal agents, so it's necessary but not sufficient to have information about them. I suggest the following approach:

1. Investigate (i.e., do a literature search on) possible time-varying causes of sleep disturbance.
2. Collect months-long time series data on these potential causal factors, concurrent with time series data on your sleep patterns.
3. During your waking hours, study causal time series analysis.
4. Analyze your data and intervene on any identified causes.
5. (Optional) Write it up for LW or for even as a journal article.

Comments:

On point 1, I'd say the odds are better than 50:50 that you've already done the relevant lit review. I still included it in the plan of attack in case you hadn't.

On point 2, I suggest months-long exhaustive data collection only because you've already shown that you are motivated to do it. Also, I think it's important to collect data on your eating habits whether or not it shows up as potentially causal in the lit review.

Discussion seems like the correct place for this post to me.

Is there some sort of pattern detecting thing ... that could automatically take a time series data and predict the next values based on an unknown, complex model?

There are lots of tools like this. The problem is getting enough clean data. If you don't have enough data, you won't be able to use a complex model without overfitting. If the underlying phenomena (i.e. your sleep cycles) is itself complex, then you won't be able to obtain a good description. On the other hand, it can't hurt to try. How much data do you have?

You could ask this guy. I'm saying nothing of the "worthiness" of credentials, just that he's studied and written quite a bit about sleep. Here's an article HERE.

He advocates for biphasic sleep (longer 7-9hr "core" + ~30min nap) and thinks it's how the body prefers to sleep.

Maybe write to him and include your data and see if he can offer you any input? He may very well get contacted by people in your situation and have some information about what helped them? I do know that he advocates for people having trouble to attempt "free running sleep" if possible -- basically sleeping when tired, waking when not. Obviously this is impractical for most, but if your situation allowed for it, you could really give a shot to shutting out distractions and just sleep and keep a log.

Anyway, not much of a help, but I stumbled on his work a bit back and thought I'd at least toss it out as a suggestion. At least contacting him might prove helpful. See his "Apology" link near his contact information. He sounds tough to get a hold of. I'd make it a "provocative" subject line, like "Need serious sleep assistance. 5 mos of data attached."

What information are you hoping to get out of this? "I just slept for 7 hours, I can expect to be awake for X hours with probability pX"? Or "my sleep patterns repeat this 125 hour cycle"?

If you have the data in a convenient format, I could take a look at it more easily than I could explain how to take a look at it.

Not really an answer to your specific question, but have you tried, or considered trying, something radically different, like a polyphasic sleep cycle (e.g. a 20-minute nap every four hours and nothing else)?

If you want a model, how about this for a simple but not very good first attempt:

• markov process, with each state an ordered pair, where the first element is the time you went to sleep, and the second the duration of your sleep.

I find it impossible to wake up at a consistent time every day (+/- 8 hours)

By this, do you mean that the time you wake up varies over a full range of 16 hours around its median?

If it's really that bad, you should use a mild sedative to force yourself to fall asleep at a consistent time.

I'm aware that you're asking how to predict, rather than how to control. Unfortunately the ideas I have on hand are about how to control.

First there's Ferris' hack of taking a cold bath, item 2 of:

http://www.fourhourworkweek.com/blog/2008/01/27/relax-like-a-pro-5-steps-to-hacking-your-sleep/

I have not tried that myself. Second, there's orange glasses:

http://www.cleveland.com/healthfit/index.ssf/2010/06/goggles_with_orange_lenses_blo.html

These seem good-enough and they're cheap:

http://www.amazon.com/gp/product/B0001YXG8S

My own self-experimentation says they seem to help. Wear them when it's dark outside so artificial light doesn't push around your sleep schedule.

I'd start by playing around with the data, trying to find different ways to organize it, identifying and defining relevant variables, and making graphs to look for patterns. Then you can start to look at relatively simple tests of relationships, and once you have a better idea of how the data are structured and what sorts of relationships seem to be present then you can try fancier math (if necessary).

For instance, you linked to your data organized by sleep cycle, starting a new row of data each time you fell asleep. I made a spreadsheet here using that organization (but slightly different formatting which was easier for me to work with) and used it to make this graph, which shows how much time you slept in blue and then the following amount of time you spent awake in orange, and then starts the next sleep period on the row below it. Two things that jump out from the graph are that you have lots of short cycles (under 5 hours, even) and that most of the cycles are under 24 hours. If you tend to find yourself on a cycle of over 24 hours (going to bed later and later each day), that must be from sleeping multiple times in a single calendar day.

Another way to organize the data is by calendar day. You could have one row for each calendar day (possibly starting each day at a time other than midnight, like the time when you are most often awake or when you're most often asleep) and make a two-color graph of when you're asleep & awake, like the one I linked to (except this time it would be a rectangle). With the data organized that way, you could look at questions like: how often am I awake at each time of day? If I am awake at a certain moment, how likely am I to be awake 24 hours later? If I am asleep? Or x hours later - you could make a graph where the x-axis is the number of hours later, and the y-axis is the probability of being awake that long later (give that you're awake/asleep at t0). How does this change depending on time of day?

There are also various correlations that you could look at, or graphs that you can plot of 2 variables, like bedtime, length of sleep, wakeup time, length of awakenness, sleep debt (percent of time asleep during previous 72 hours? during previous 4 sleep cycles?). You might need to do something about the short cycles (leave them out? combine them with adjacent sleep/wake periods?). Here is a graph of amount of sleep vs. bedtime, which seems to show a pattern (you sleep longer when you fall asleep earlier) if we ignore the short sleeps.

Or, come up with some definition of a "normal" sleep cycle (e.g. one lasting close to 24 hours, which includes close to 8 hours of sleep) or a normal day (e.g. sleeping at least 6 hours from 10pm-10am and then being awake for at least 9 hours from 10am-10pm). How common are these "normal" cycles/days, are there any patterns to when they occur, do they come in streaks, and what seems to happen to precipitate the end of a streak of normality?

There are a bunch of other things like these that you could try. If you've already done some, you could share them here.

Have you been collecting the data? There are all sorts of fun things you can do to make predictions but the hard part is collecting it in the first place. Sleep times, wake times. Then start adding in potentially relevant information - like food, exercise, subjective feelings of fatigue and social events. Extracting a pattern after that tends to be easy.

I think you're taking the fundamentally wrong approach. Rather than trying to simply predict when you'll be sleepy in the near-term, you should try to actively get your sleeping patterns under control.