So, we have at least “one clue”, but also “no experience” from which we can draw a conclusion? And “predicting” things about the future is “seriously dubious”, but “propos[ing]” expectations about the future is not? Yikes dude, that’s some fucking sophistry.
I disagree with this. In many fields, we can make reasonable predictions about things we have not experienced directly, by reasoning on what we do know about the components. We predicted that radioactive materials would create a nuclear chain reaction before we first tested a nuclear bomb, because we had a theory of how such a reaction would work.
Similarly, we have good evidence of the constraints of consciousness and information processing, and we can make predictions about the limits of conscious systems with a not-insignificant degree of confidence. We can reason about how small a system could be that can compute a certain algorithm in a certain amount of time, based on the minimum amount of heat we have good reason to expect such information processing to produce. We can estimate the minimum calculations per second that consciousness will require, and the minimum energy requirements of such a system. With things like these, we can reach conclusions like “superintelligent AI likely can’t run on a Pentium II”, and thereby constrain what a superintelligent AI is likely to be able to do (e.g. copy itself to a system running on a Pentium II).
None of this is 100%, but it’s a damn sight more than 0%. We can constrain our expectation based not only our experience of smarter organisms interacting with less smart organisms, but also on what we have reason to believe constrains smarts.
Clues abound.