"A system that collects about 100 million questionnaire data on SNS, inserts it into a DB (database), and aggregates, can be created on even my personal computer at home"

"The former Governor of Tokyo, who is not an IT engineer, would have spoken without knowing the details."

Probably, I think that this kind of system construction can be done in one day.

Such as recording about 100 million records, can be processed by a script language and the data aggregation can be completed in a few minutes by a single operator.

-----

This time, the Ministry of Health, Labor and Welfare implemented

"A nationwide survey conducted by the Ministry of Health, Labor and Welfare using the free communication application "LINE" to combat new coronavirus"

is "unusual jackpot system", I think.

Because, this system

can process data for users, as well as system managers, quickly, cheaply, and easily.

can secure anonymity.

is easy to collect meaningful answers (uninterested humans will ignore it if they take time to enter wrong answers)

In this way, this system can collect data of tens of millions of people at a low cost. Such a “delicious system” is rare.

-----

By the way, I ran a simulation of 300 million people on my home computer 10,000 times in a row, and it took about 14 hours.

In other words, data processing for 300 million people is completed in about 0.5 seconds.

If someone ask me whether memory writing by simulation can be considered as similar to DB writing, I can say that recent DB writing may be faster than memory writing (It is true)

Writing and searching with 100 million DBs by LINE this time could have taken a few seconds if the total CPU time was used.

-----

The former Governor of Tokyo may have had the image of "several dozens of people print data results and hit calculators" in data aggregation.

But that image is not correct.

"Aggregation operations and documentation were completed within a few hours by one engineer."

I estimated that.

That's because even an engineer, like me, can complete about 100 million lines of SQL processing in this time.

The performance of big data and SNS message processing these days has sometimes surprised me even on site.

So his remarks could not have been helped.

-----

Nevertheless, if he is saying something that is not his specialty, it is better to spend a little time researching it himself or asking an expert before speaking.

We should be especially careful when criticizing something or someone.

I write this diary with my self-awareness.