[ANN][XEL] Elastic Project - The Decentralized Supercomputer

[Evil-Knievel]

Feature Announcement / Discussion

Currently, Elastic only supports "bruteforceable" work and with it it covers a lot of work types. However, this scheme does not allow specific use cases where work is performed on a large portion of data and does not operate on a "search space" (like elastic's 384 bit long random entropy) but rather on different portions of blob data. Typically an example might be the rendering of a large video or audio stream.

So I suggest to add a second type of job type which works a bit diffently.
Similar to Zennet you can in addition to the already present "ElasticPL" scheme, write "Elastic Dist" programs. Here, a work author creates a regular job but does not submit source code to the blockchain but rather submits a "IP for a public Elastic Dist node". Miners then connect to the node which then distributes both "Data and Sourcecode" which has to be then executed by the miners. Here, we again work ith the safe-to-execute ElasticPL language.

Here, the "Elastic Dist Node" can seperate the work itself into packages, and submit them along with data (up to x MB) to the nodes. They return the result back and get paid the PoW price. Here, we cannot use any proof of execution in order to prevent blockchain cluttering, but we use (like Zennet does, big credits to them) micropayments. Miners do not execute other tasks until they receive the micropayment from the last code portion.

So let's say we have 2 miners, and a Elastic Dist job with a public work distribution node. Imagine we want to render a video. The job is split into 100 different sub-tasks (which fit in the maximum allowed WCET per program). The work author announces this job and says it pays 100 XEL per execution. The 2 miners then poll 1 subpackage each, calculate it and submit it to the Elastic Dict node and wait for the micropayment of 100 XEL. Once received, they poll the next subpackages. If not, they wasted at most WCET computation time.
If the Elastic Dist node becomes unreachable, no ork is done and no XEL is paid. The job just times out.

This opens a few attack vectors of course were the Elastic Dist node does not send any micropayment. We would have to think about it, how to mitigate that.

This scheme would be a bit trust based, but in conjunction with the traditional ElasticPL work type, allow to do ANYTING with elastic ... from mining, bruteforcing up to rendering arbitrary tasks.

For those who need more security write their programs in the somewhat limited Elastic PL scheme, those with special requirements use Elastic Dist which is somewhat trust based.

What do you think, ttookk and others?

[1714023352]

1714023352

[ttookk]

I like it.

I see this as a good way to make use of the network, when there are no "technical" problems to be solved. As pointed out earlier, "voids" in the workload might become a problem when they get too big.

I think, rendering a video would be one of the most common use cases, in this youtube era, where spitting out quality content as fast as possible is a must to stay over water. Now, this might not be the userbase, Elastic wants to attract, but why not? As long as they are paying (and telling their viewerbase about Elastic). Effectively, this might bring more popularity to and more computing power into the Elastic network, which is good for all sides involved.

I wrote this part like a thousand times, but still: this may be the point, where we should think about a reputation system. See my previous posts and all that.

By the way, it is somehow possible to encrypt a video or other file in a way, that it can still be rendered but is unreadable for the miner?

These are my initial thoughts, and I am more than willing to step back from them.

[Mrboot]

This is a hard project to follow if your not technical at all, but damn it looks great.

Im really impressed and happy i invested, but now i have a question.

I cant find any details about the chain anymore, is it still POS and what are the rewards.
Small things like that.

And developers if this work out i will def send you some nice extra working tip.

Kind Regards,

[ttookk]

(…)

I cant find any details about the chain anymore, is it still POS and what are the rewards.
Small things like that.

(…)

I'm a little fuzzy regarding this part, but as far as I know, it still has PoS. Since the number of tokens is fixed, the PoS reward would be transaction fees only.

[ttookk]

What may create a little problem, though, is the question of how to value the two types of jobs relative to each other. I mean, there is the risk that one of the two job types is that much more profitable, that miners won't bother to work on the other. You could hope for "the market" to sort it out, i.e. the pricing will be adjusted according to it, but it might not be enough.

Here are some random thoughts I had today: I was thinking about how Elastic will be used and will be perceived of the general public and the more I think about it, the more I think that its place is more of a backend solution than a full-blown, frontend flashy GUI thing. With the two jobs in place, we will probably have two main types of job authors:

1.: Scientific users, who need numbers to be crunched and problems to be bruteforced. Those will probably write their own code, or are otherwise able to use and interpret existing code for their tasts. It is safe to assume, that they don't need a shiny frontend, they need something reliable to get the job done.

2.: Everyday users, that need big chunks of data, e.g. a video, processed. Let's assume, there is a youtuber who wants to put out quality content in a timely manner. They probably don't have a renderfarm at hand, so they want to use Elastic for the job. For them, there's gonna be a number of obstacles:

- I wouldn't expect them to be even able to use the terminal, or console. They would have to use premade code that fits their requirements and would probably need a easy to use GUI that tells them exactly what to do. To reach them, some kind of database is needed(to be clear, I don't mean that it's the Elastic core teams job to create such libraries, but it would be handy if someone did at some point. As with other blockchains, once the whole thing takes off, I'd expect multiple clients to pop up that maybe are specialized for specific use cases. If rendering videos is all you're gonna do with Elastic, a client that is helping you with that and nothing else would be perfect.)

- Bitcoin itself is still a barrier to entry, the need to purchase XEL will be even worse. What Elastic would need is a gateway, where you can buy the exact amount of XEL needed, no questions asked. On the frontend, it would look like you paid with BTC or even fiat, the XEL is only used in the background, nearly invisible to "regular customers". It may even be possible, that they use clients to rent computing power without realizing, that they actually use blockchain technology.

- Volatility is a general problem crypto has and I believe it is one of the main obstacles for mass adoption. Let's say a bread costs 3 €. If you'll ask people on the street, whether they would like to know that bread costs 3 € tomorrow as well, or you'd give them a 33/33/33% chance, that a bread will cost 1,5 €, 3 € or 4 € tomorrow, I think most will opt for stability, although in the long run, taking that chance would give them a cheaper price. Now, it is hard to adjust Elastic in a way, that volatility is kept at a minimum, but I suspect, that the volatility will keep a lot of potential users away.

If we want Elastic to be adopted by a mainstream group (I'm not saying that we want to, but let's assume we do), we have to make the entry as simple as possible. This includes:
- An easy-to-use GUI.
- Getting rid of the necessitiy to write own code and use "off the shelf" solutions instead.
- Creating a gateway, that safely calculates and purchases the right amount of XEL in the background.

This is not something that needs to happen NOW. I think it would be perfectly fine to release Elastic into the wild with the one type of job as a possibility, then work on the other part. Depending on its success, it might attract other people and devs, who will work on the second type of job and hopefully develop aforementioned clients.

As I said before, I don't expect this to be the job of the core team, to create easy-to-use solutions for non-technical users, but I think the question whether we expect other clients with premade preferences to emerge or not could possibly influence how Elastic is developed.

[akhavr]

Miners do not execute other tasks until they receive the micropayment from the last code portion.

Any other task at all or any other task from this work distribution node?

So let's say we have 2 miners, and a Elastic Dist job with a public work distribution node. Imagine we want to render a video. The job is split into 100 different sub-tasks (which fit in the maximum allowed WCET per program). The work author announces this job and says it pays 100 XEL per execution. The 2 miners then poll 1 subpackage each, calculate it and submit it to the Elastic Dict node and wait for the micropayment of 100 XEL. Once received, they poll the next subpackages. If not, they wasted at most WCET computation time.

How I, as a work author, can be sure that the work is really performed and I've got not just a noise?  Running the same job on 2+ miners?  Miners reputation?

[akhavr]

2.: Everyday users, that need big chunks of data, e.g. a video, processed. Let's assume, there is a youtuber who wants to put out quality content in a timely manner. They probably don't have a renderfarm at hand, so they want to use Elastic for the job. For them, there's gonna be a number of obstacles:

Such users be better served by regular web apps.  Say, I'd create a site, where you can upload your video and get it rendered for a paypal payment.  The fiat price would include all kinds of risks -  volatility, chargebacks - and some profit of course.

[tomkat]

Miners do not execute other tasks until they receive the micropayment from the last code portion.

Any other task at all or any other task from this work distribution node?

So let's say we have 2 miners, and a Elastic Dist job with a public work distribution node. Imagine we want to render a video. The job is split into 100 different sub-tasks (which fit in the maximum allowed WCET per program). The work author announces this job and says it pays 100 XEL per execution. The 2 miners then poll 1 subpackage each, calculate it and submit it to the Elastic Dict node and wait for the micropayment of 100 XEL. Once received, they poll the next subpackages. If not, they wasted at most WCET computation time.

How I, as a work author, can be sure that the work is really performed and I've got not just a noise?  Running the same job on 2+ miners?  Miners reputation?

I guess, the miners would need to run an algo/a script/whatever is supposed to do the rendering job, and not just a series of computations not related to input data.

[Evil-Knievel]

Miners do not execute other tasks until they receive the micropayment from the last code portion.

Any other task at all or any other task from this work distribution node?

So let's say we have 2 miners, and a Elastic Dist job with a public work distribution node. Imagine we want to render a video. The job is split into 100 different sub-tasks (which fit in the maximum allowed WCET per program). The work author announces this job and says it pays 100 XEL per execution. The 2 miners then poll 1 subpackage each, calculate it and submit it to the Elastic Dict node and wait for the micropayment of 100 XEL. Once received, they poll the next subpackages. If not, they wasted at most WCET computation time.

How I, as a work author, can be sure that the work is really performed and I've got not just a noise?  Running the same job on 2+ miners?  Miners reputation?

The second scheme was just a first idea, open for discussion ;-) And you have perfectly argued that probably such tasks (as rendering) should be performed on centralized services.

But we can brainstorm a little, to identify if its really so bad and we should leave it out, or if it's worth pursuing this idea.

To your first question: Only for the work distribution node (or any other that the miner is connected to). The blockchain would not track any work, it would be used only to negotiate work between "distribution node" and "worker node" and keep them working by periodically sending micropayments.

To your second question: of course you canot be sure, this is of course a problem and this is why i personally think Zennet is inferior to us. You have to "trust" or somehow ensure that the result is plausible or not yourself. If you think the result is bad, you can deny the micropayment and not distribute any work to the connected worker node anymore.

This scheme is of course fragile! I am just interested if it could somehow work as an "additional work type" or if we better leave it out and stick to our "verifiable computation scheme" which is robust.

[cryptoboy.architect]

Love the project development so far. I was looking into building something similar.

Few ideas here:

- instead of inventing a new language, why not use OpenCL? It is fairly straightforward and kernels compile on the host. As an added benefit - you can request work done on the GPU. Now we are talking real computing power.

- hook to IPFS, so that a worker that requires big data could fetch it by a hash. Result of the work, if it is large could also be returned via IPFS hash.

- with respect to verifying the work - just like in mining, a job could have a "manager" script, which computes portions of the work. i.e. it's the developer's responsibility to provide some ratings metrics, so that job can be assigned to the most reputable hosts.

[ttookk]

(…)

How I, as a work author, can be sure that the work is really performed and I've got not just a noise?  Running the same job on 2+ miners?  Miners reputation?

The second scheme was just a first idea, open for discussion ;-) And you have perfectly argued that probably such tasks (as rendering) should be performed on centralized services.

This might be an answer to the trust and possible abuse scenario:

- with respect to verifying the work - just like in mining, a job could have a "manager" script, which computes portions of the work. i.e. it's the developer's responsibility to provide some ratings metrics, so that job can be assigned to the most reputable hosts.

Other miners recalculate small random portions of the work and check whether they fit or not.

But we can brainstorm a little, to identify if its really so bad and we should leave it out, or if it's worth pursuing this idea.

(…)
This scheme is of course fragile! I am just interested if it could somehow work as an "additional work type" or if we better leave it out and stick to our "verifiable computation scheme" which is robust.

I still like the idea, but as pointed out, I'm not sure whether it is something you should pursue right now, maybe this should be treated more like a possible future addition to Elastics main functionality.

[wwdz99]

The OP has been updated with the current website.

Well done Lannister ,Nice to see u active again!

[mr.coinzy]

Did anyone hear about the current problems with blockchain.info?
Seems there are issues there and I really hope accounts where not compromised and such...
What will happen if we can not safely access blockchain.info accounts? some people sent their btc to the funding from such account (myself included).
Thanks.

[anhpt192]

Did anyone hear about the current problems with blockchain.info?
Seems there are issues there and I really hope accounts where not compromised and such...
What will happen if we can not safely access blockchain.info accounts? some people sent their btc to the funding from such account (myself included).
Thanks.

I tested some BTC transactions from exchange to exchange, transfers were done and coin was credited to my balance.
I don't think these is a big issue.

[ttookk]

Did anyone hear about the current problems with blockchain.info?
Seems there are issues there and I really hope accounts where not compromised and such...
What will happen if we can not safely access blockchain.info accounts? some people sent their btc to the funding from such account (myself included).
Thanks.

I'm pretty sure there is a way to download your private key from your blockchain account. If you are afraid, that you won't have access to blockchain.info, do it(might be a good idea anyway), store it in a safe place and you can access your BTC address through pretty much any other wallet, that allows import of a priv.key (which are probably all of them).

Regarding video rendering on Elastic: The name of the dedicated client should be

XELLULOID

Had to be done

[tomkat]

Did anyone hear about the current problems with blockchain.info?
Seems there are issues there and I really hope accounts where not compromised and such...
What will happen if we can not safely access blockchain.info accounts? some people sent their btc to the funding from such account (myself included).
Thanks.

They claim it's DNS issue - see here https://twitter.com/blockchain.
I don't know if such issue require several hours to resolve though.

You need your private key(s) to claim XEL, so if you haven't got it from there yet, then you can only wait

[Evil-Knievel]

Did anyone hear about the current problems with blockchain.info?
Seems there are issues there and I really hope accounts where not compromised and such...
What will happen if we can not safely access blockchain.info accounts? some people sent their btc to the funding from such account (myself included).
Thanks.

They claim it's DNS issue - see here https://twitter.com/blockchain.
I don't know if such issue require several hours to resolve though.

You need your private key(s) to claim XEL, so if you haven't got it from there yet, then you can only wait

Or use one of the BIP39/BIP44 converters (e.g. https://iancoleman.github.io/bip39/) to convert your blockchain.info mnemonic seed to a private key stream.

Disclaimer: The linked project above is not mine and I have not reviewed it, just found it on google ... use at own risk and check the source code first. Ideally, use on an offline computer!!!

[akhavr]

To your first question: Only for the work distribution node (or any other that the miner is connected to). The blockchain would not track any work, it would be used only to negotiate work between "distribution node" and "worker node" and keep them working by periodically sending micropayments.

Tnx

To your second question: of course you canot be sure, this is of course a problem and this is why i personally think Zennet is inferior to us. You have to "trust" or somehow ensure that the result is plausible or not yourself. If you think the result is bad, you can deny the micropayment and not distribute any work to the connected worker node anymore.

This scheme is of course fragile! I am just interested if it could somehow work as an "additional work type" or if we better leave it out and stick to our "verifiable computation scheme" which is robust.

It's good to have this work type, my point is that the network has to have rules to help to set up such work.   

In this context I like proposal by @ttookk, even if the ratings update would be a job's point of responsibility.  OTOH then we'd have to have something to mitigate attacks on rating by malicious job authors.

+1 to the future addition.

[Evil-Knievel]

@akhavr,ttook: The problem with rating schemes is imho that you can create an infinite number of alternative identities, all with a clear background. Reputation may work in centralized systems that require some sort of authentication like e.g. eBay, but if we do not want to penalize new users without any rating, we will have a hard time coming up with a reputation system that really works. I mean I could rip off some workers, collect negative reputation, move the XEL to a fresh account and start over.

What do you think?

[Evil-Knievel]

Love the project development so far. I was looking into building something similar.

Few ideas here:

- instead of inventing a new language, why not use OpenCL? It is fairly straightforward and kernels compile on the host. As an added benefit - you can request work done on the GPU.

My thoughts on this:

- In order to know how long a program will execute (this is needed for our difficulty retargeting mechanism), it's required to perform a worst case execution time (WCET) analysis. A WCET execution analysis on programs written in traditional programming languages such as C, Java, Python or OpenCL is hard to achieve, if not impossible. The reason for example are loops (which cannot be estimated how often they will be run unless actually executing the program, and then the iteration count can depend on other factors such as the input itself).

- In the common case you can allocate as much memory as you like … this can cause out of memory errors on less potent hardware while going through on good hardware → the network can be split by shooting down certain nodes keeping alive others.

- Ususally, it is possible to create programs that will run forever and you cannot decide that beforehand for the project to reject such software (it's basically the halting problem). This could turn all nodes into a heavy, hot paperweight.

- Programs written in other programming languages can crash (division by zero, null pointer exception, etc.) and can be used to attack the network.

This, and some other reasons, require the creation of a new language where the above problems do not occur in the first place. Our language does not crash, it always terminates, the run time can be estimated beforehand and you can only use a limited amount of memory allowing us to give minimal system requirements where we can guarantee that every program will execute successfully if a machine matches those system requirements.

I have added this to http://elastic-project.com/elasticpl

Now we are talking real computing power.

You can transform any ElasticPL program to an equivalent OpenCL program, it's just not the case vice versa.