Some more thoughts: probably the easiest way to get validation for this theory is to build a state-of-the-art chatbot using a dead simple architecture. If the theory is as good as I think it is, this should lead to a terrifyingly good chatbot that will (in the limit over time) approach passing the Turing test.

I recommend using the tensor2tensor library with the transformer model and byte-pair encoding. The chatbot should be hooked up to a public-facing website such that people can converse with it and can reward or punish the chatbot whenever they like or do not like what it says. Since people are terrible, we will need to have some sort of trusted set of people who can log in with a password and chat with it; these "known good" interlocutors can dish out rewards that are of much higher value than that of randos. In fact, it might be the case that the reward signal from randos should be compared to the known-good reward signal and should be flipped if the system thinks the rando is trying to train the chatbot to say offensive shit.

Training data should be fed in in batches over a period of perhaps one hour, and then replaced with the next batch when the hour is up. Data should simply be transcripts... need to think about this more, but essentially the task we want to train the system on is to produce the next line of dialogue given the entire previous conversation. There is some work to do to figure out how to encode all of this in a way that works with tensor2tensor's existing shit without having to reinvent the motherfucking wheel all the time.