Definitely an interesting use of the tech- though the capability needed for that to be a really effective use case doesn't seem to be there quite yet.

When editing down an argument, what you really want to do is get rid of tangents and focus on addressing potential cruxes of disagreement as succinctly as possible. GPT4 doesn't yet have the world model needed to distinguish a novel argument's load-bearing parts from those that can be streamlined away, and it can't reliably anticipate the sort of objections a novel argument needs to address. For example, in an argument like this one, you want to address why you think entropy would impose a limit near the human level rather than at a much higher level, while listing different kinds of entropy and computational limit aren't really central.

Also, flowery language in writing like this is really something that needs to be earned by the argument- like building a prototype machine and then finishing it off with some bits of decoration. ChatGPT can't actually tell whether the machine is working or not, so it just sort of bolts on flowery language (which has a very distinctive style) randomly.

I only had to see this:

In the looming shadow

to know it was ChatGPT.

...of an impending AI superintelligence

Aaaaaaah!

...society oscillates between hope and trepidation.

o god make it stop make it stop

If you write a long stream of consciousness draft, ChatGPT will not turn it into a short, concise expression of your thought. That can only come by working on it yourself.