[MERGED] BIP-39 List of words in Portuguese accepted!!

[NotATether]

Someone just made a pull request for a Romanian wordlist: https://github.com/bitcoin/bips/pull/993

That should be a wake up call for me to speed up development. I almost finished the programmatic API. Docs are easy to write, and I think I'll use Nose as my testing suite.

EDIT: Levenshtein distance section of the API completed, initial unique characters and maximum length are almost finished


What would you guys think of me adding a method to fetch the wordlist from the internet, by supplying the URL? It would allow you to instantly check the wordlist if it's on places like Github without having to download it first.

e.g. In your web browser you go to the wordlist file on Github, and then click on View Raw to get this kind of link https://raw.githubusercontent.com/bitcoin/bips/03ff98d00717804723a2c4db8188c0b5cf0cbfbf/bip-0039/romanian.txt, which is a plain text file which can be downloaded and processed easily.

[1713400264]

1713400264

[1713400264]

1713400264

[bitmover]

What would you guys think of me adding a method to fetch the wordlist from the internet, by supplying the URL? It would allow you to instantly check the wordlist if it's on places like Github without having to download it first.

e.g. In your web browser you go to the wordlist file on Github, and then click on View Raw to get this kind of link https://raw.githubusercontent.com/bitcoin/bips/03ff98d00717804723a2c4db8188c0b5cf0cbfbf/bip-0039/romanian.txt, which is a plain text file which can be downloaded and processed easily.

You can add a field to upload the file directly or to add the URL. This makes little difference, as the person who is working with the wordlist certainly have the txt file in his computer or he can access github directly.

It is more important to be able to upload the txt file imo

[NotATether]

What would you guys think of me adding a method to fetch the wordlist from the internet, by supplying the URL? It would allow you to instantly check the wordlist if it's on places like Github without having to download it first.

e.g. In your web browser you go to the wordlist file on Github, and then click on View Raw to get this kind of link https://raw.githubusercontent.com/bitcoin/bips/03ff98d00717804723a2c4db8188c0b5cf0cbfbf/bip-0039/romanian.txt, which is a plain text file which can be downloaded and processed easily.

You can add a field to upload the file directly or to add the URL. This makes little difference, as the person who is working with the wordlist certainly have the txt file in his computer or he can access github directly.

It is more important to be able to upload the txt file imo

Where are likely sites where people would want to upload the wordlist though? At least for Github and other source control sites, it can already be tracked and committed with git, so I don't really see a point in adding such a feature.

[NotATether]

Small update, in case you thought my initiative stagnated.

I am currently writing documentation for my validator library. Once I finish that, I'll make a release candidate and put it on Github for you guys to test. I'm anticipating a proper release within 1 or 2 weeks.

[bitmover]

I am going to make an updated as well
We submitted our wordlist as a pull request here:
https://github.com/bitcoin/bips/pull/998

But no answer from contributors who could approve it. Any ideas?

[Coding Enthusiast]

https://github.com/bitcoin/bips/pull/998

You should squash your 151 commits into 1.

But no answer from contributors who could approve it. Any ideas?

Sometimes PR on BIPs repo go unnoticed for very long times. Last commit to word-lists was merged by luke-jr, try mentioning their username under your PR.

[TryNinja]

You should squash your 151 commits into 1.

It's done. I would appreciate if you can confirm I didn't make any mistake. It's the first time I do that.

https://github.com/bitcoin/bips/pull/998

[Coding Enthusiast]

It's done. I would appreciate if you can confirm I didn't make any mistake. It's the first time I do that.
https://github.com/bitcoin/bips/pull/998

Looks fine to me. You can also see the actual change in Files tab to see if everything is alright.

[NotATether]

As promised here is version 1.0.0rc1 of BIP39 Validator. Please make sure you clone the release-1.0.0rc1 branch. You all are welcome to test it and check for bugs but be warned that this is alpha quality software and the API and main program may not even work and throw trivial syntax exceptions.

@bitmover: here's the page you can download source code from: https://github.com/ZenulAbidin/bip39validator/releases

I haven't written the unit tests yet, I should do so sometime soon to at least make the program runnable.

Note: I heavily borrowed README formatting from a completely unrelated project called nanogui so there may be broken links in my project pointing to resources there. This will be fixed soon. Screenshot also needs to be replaced preferably with one from a live run of BIP39 Validator.

[bitmover]

As promised here is version 1.0.0rc1 of BIP39 Validator. Please make sure you clone the release-1.0.0rc1 branch. You all are welcome to test it and check for bugs but be warned that this is alpha quality software and the API and main program may not even work and throw trivial syntax exceptions.

@bitmover: here's the page you can download source code from: https://github.com/ZenulAbidin/bip39validator/releases

I haven't written the unit tests yet, I should do so sometime soon to at least make the program runnable.

Note: I heavily borrowed README formatting from a completely unrelated project called nanogui so there may be broken links in my project pointing to resources there. This will be fixed soon. Screenshot also needs to be replaced preferably with one from a live run of BIP39 Validator.

Hello NotATether,

I tried to test your software, but I had some problems. I will give you a few suggestions:

1- I think you should make an ANN thread for it.
2- This link is broken (documentation) https://bip39validator.readthedocs.io/
3 - I was not able to install it, saw this message:

Code:

pip install bip39validator
ERROR: Could not find a version that satisfies the requirement bip39validator (from versions: none)
ERROR: No matching distribution found for bip39validator

There was no clear message about what are the requirements which are not installed. Personally, I don't like to install too many python libraries because they can mess up with my environments. However, I tried to create in both my testing env and my base env, same error.

I think Python lacks a good way to share software. I would be nice if you could create a standalone version of your program, or even an online web version (you just send your wordlist.txt file to a server and it returns your python output). I don't know if it can be done, but i think it can be done with Ruby.
You are also supposing that people who create lists have anaconda installed, which is a 400mb software and not newbie-friendly. In my group of the 10 people who created the portuguese wordlist, only I (afaik) use python.

There are also some libraries which allow you to create a standalone version of your code.However they do not work very well, AFAIK. Once I created one of those standalones which had nearly 300mb (basically all anaconda package)

[NotATether]

It's going to be quite a challenge to compose this on mobile, but I hope it comes out correctly. One draft already got trashed by Android so I had to type this again (on mobile again as I'm not available yet on desktop)

Hello NotATether,

I tried to test your software, but I had some problems. I will give you a few suggestions:

1- I think you should make an ANN thread for it.
2- This link is broken (documentation) https://bip39validator.readthedocs.io/
3 - I was not able to install it, saw this message:

Code:

pip install bip39validator
ERROR: Could not find a version that satisfies the requirement bip39validator (from versions: none)
ERROR: No matching distribution found for bip39validator

First of all thank you for taking the time to try to run this, I really appreciate it especially since it's difficult to find collaborators for small projects.

Yes there are a ton of things which are currently broken, I need to get around testing the library sometime tonight.

Testing has been constantly delayed by continuous power outages etc. so I'm in the final stages of leasing a dedicated server for staging my development on so I don't have to reopen my applications each time.

I have not pushed bip39validator to PyPI or Readthedocs yet, The Readthedocs listing I hope I can do that tonight, as for the PyPI release I am waiting until I publish a stable release before I publish it. I don't want to expose development versions of libraries to PyPI, because people expect them to just work. For now the only way to get the source is to download the source code from the Releases page on Github, or by cloning the release-1.0.0rc1 branch (the other branches are subject to breaking changes).

Then you'd run the main.py script with arguments as I'm not sure if my setup.py script from setuptools works yet. Do not install it using pip, it won't work until I push it to PyPI.

There was no clear message about what are the requirements which are not installed. Personally, I don't like to install too many python libraries because they can mess up with my environments. However, I tried to create in both my testing env and my base env, same error.

bip39validator only has three dependencies: jellyfish to compute Levenshtein distances, requests to support downloading wordlists from URLs, and rich for formatting console output. The list of dependencies can also be viewed in the requirements.txt file (it's meant to be read by pip to automatically install dependencies).

I always test libraries in virtualvenvs and not on the base python installation because the setup.py logic might be broken and need tobbe debugged. No one wants to deal with a broken library that won't install in their main env, when they can simply delete a virtualenv, create a new one and start over.

I think Python lacks a good way to share software. I would be nice if you could create a standalone version of your program, or even an online web version (you just send your wordlist.txt file to a server and it returns your python output). I don't know if it can be done, but i think it can be done with Ruby.

Setuptools can make a standalone program out of python programs, if you give it the file that's supposed to represent the main script. Then you can just run it like a shell command. I just haven't got around implementing that yet.

Maybe I'll develop a GUI version as well, which primarily benefits Windows users who don't want to run programs from Command Prompt. But no more features will be added to this version, so that will have to wait until 1.1.0.


I could also add self-hosting support using Django so people can subit wordlists to a server that does the validation for them, that'll allow me to advertise this to wordlist drafters who don't want to install Python. It will have to wait until the next major version though, because of it's complexity. Thank you for this suggestion, @bitmover (and I even figured this out only after writing this a second time! Some things stick to your head much later.)

For now though, I will host a Jupyter notebook out of my own pocket so at least you guys can experience a demo version without installing it.

I put several man-hours in this project already so it's too late to change languages  and frameworks.

You are also supposing that people who create lists have anaconda installed, which is a 400mb software and not newbie-friendly. In my group of the 10 people who created the portuguese wordlist, only I (afaik) use python.

There are also some libraries which allow you to create a standalone version of your code.However they do not work very well, AFAIK. Once I created one of those standalones which had nearly 300mb (basically all anaconda package)

This project does not use Anaconda and you don't need it to install this. It only requires the three dependencies I mentioned above, jellyfish, requests and rich.


Urgent TODO list:

- Finish setup.py, so people can at least install it sanely.
- Run main.py and make sure there are no basic runtime errors
- Push documentation to Readthedocs Done, please go to https://bip39validator.readthedocs.io (though it's missing all the docstrings, I need to fix that)
- Create unit tests and test wordlists to find bugs in edge cases

Then we can talk about releasing rc2.


I'm not making another post just to announce this, but you all can view progress of this at

https://github.com/ZenulAbidin/bip39validator/projects/1

and

https://github.com/ZenulAbidin/bip39validator/milestone/1

(Github project and milestone dashboards respectively)

There are only three bugs/enhancements I need to attend to before I mark another release candidate.

[NotATether]

BIP39 Validator version 1.0.0rc2 released!

This took way too longer than it should have, but now that the heavy work is done, only a few finishing touches are left to make and then the stable version 1.0.0 should be ready. I weeded most of the bugs out from this version, so you can get the source code and run python setup.py install to install bip39validator to try it out yourselves.

Pre-release version of BIP39 Validator 1.0.0. Much more stable than 1.0.0rc1 and should run without errors.

Notable changes in this release:

    - Setup.py now works and can be used to install bip39validator locally
    - Unit tests created
    - New documentation theme

Get the source code here: https://github.com/ZenulAbidin/bip39validator/releases/tag/1.0.0rc2

And check out the new documentation for it. https://bip39validator.readthedocs.io/en/latest/

(apologies for bumping your thread again @bitmover, this is the last pre-release I'll announce here and then I'll fork 1.0.0 to my own thread. How is your pull request doing by the way?)

[bitmover]

BIP39 Validator version 1.0.0rc2 released!

Congratulations NotATether. I will take a look at it later.

I was thinking about trying to implement your software in Javascript , in future versions. I have been studying javascript a lot, and I think it would be easier to use if you could just create a webpage where you upload your wordlist.txt and received the feedback in an HTML page as well. What do you think?

However, the levenshtein distance would be hard to implement in JS, unless if it would be possible to find a library.

(apologies for bumping your thread again @bitmover, this is the last pre-release I'll announce here and then I'll fork 1.0.0 to my own thread. How is your pull request doing by the way?)

No problem.

Our PR is still pending.
Some people answered it, but I believe our PR is not a high priority...
There is some support from some users, but still no answers from the people with the authority to merge the pull request.

https://github.com/bitcoin/bips/pull/998


Edit: Maybe you could test our wordlist in your software, and then if it passes your tool check you can make a comment? This would be also an opportunity to test and spread the word about your app.

Edit 2: Someone tested our wordlist here:
https://travis-ci.org/github/bitcoin/bips/builds/730941730

The Travis CI build. I don't know what it is, maybe something similar to your tool?

[Coding Enthusiast]

Edit 2: Someone tested our wordlist here:
https://travis-ci.org/github/bitcoin/bips/builds/730941730

The Travis CI build. I don't know what it is, maybe something similar to your tool?

It is not testing your wordlist. Tools such as travis-ci are "hosted continuous integration service" which are automatically called each time there is a new commit (or if it is set, a new pull request) and they run a script (the .yml file) that is sent to their server on each call to run. The script is usually defined to build and run tests but it can be anything.
For example I use it to build my projects on Linux-x64 and also run all my tests there; it helps catch any problems with the build or any bugs in the code that tests can catch.
In case of bips they are running this script that performs some basic checks.

[NotATether]

I was thinking about trying to implement your software in Javascript , in future versions. I have been studying javascript a lot, and I think it would be easier to use if you could just create a webpage where you upload your wordlist.txt and received the feedback in an HTML page as well. What do you think?

A webpage for uploading is on my TODO list. The beauty about modern programming languages is that they all have some sort of web framework where you can create a web app using the language you want to use. I’ll most likely use Django or Flask libraries to create the webpage which I’ll then host on my spinning rust.

Our PR is still pending.
Some people answered it, but I believe our PR is not a high priority...
There is some support from some users, but still no answers from the people with the authority to merge the pull request.

https://github.com/bitcoin/bips/pull/998

That other guy in the conversation was talking about how requests BIP39 wordlists in general don’t get merged. Many of the PRs stalled and are left without anyone actively working on replacing words. I think this is because of the sheer amount of manual verification you have to do with 2048 words, and that can overwhelm even a group of 4.

For the record, I believe the BIP39 comments are why there's a backlog of unmerged wordlists. Something about a flaw in using secure RNGs as entropy. And notice how the comment in the BIP39 standard itself says Comments-Summary: Unanimously Discourage for implementation. This comment was retroactively added on March 15, 2017 along with a bunch of other BIPs.

The problem with using a foreign wordlist, then, seems to be that while two words in different lists with the same line number correspond to each other, PBKDF2 will make a different binary seed for each wordlist because PBKDF2 uses the word characters themselves, not their line numbers in binary form. This means if you e.g. change the English words to corresponding French words, it'll treat it as a different binary seed and will make completely different private keys.

From mnemonic to seed

A user may decide to protect their mnemonic with a passphrase. If a passphrase is not present, an empty string "" is used instead.

To create a binary seed from the mnemonic, we use the PBKDF2 function with a mnemonic sentence (in UTF-8 NFKD) used as the password and the string "mnemonic" + passphrase (again in UTF-8 NFKD) used as the salt. The iteration count is set to 2048 and HMAC-SHA512 is used as the pseudo-random function. The length of the derived key is 512 bits (= 64 bytes).
~snip

I'm not even sure if maintainers are taking PRs for wordlists any more. Filtering the Github issues to the ones created after the day it was unanimously discouraged (https://github.com/bitcoin/bips/pulls?page=1&q=is%3Apr+created%3A%3E2017-03-15++bip39) show that no proposed wordlist has been merged, except for the Korean wordlist in August 2017.

Our best shot at mass adoption of foreign language wordlists, then, is if we gather them together for Electrum users like this where PRs will likely be considered.

Edit: Maybe you could test our wordlist in your software, and then if it passes your tool check you can make a comment? This would be also an opportunity to test and spread the word about your app.

I could, or you could even do it yourself by running bip39validator WORDLIST_FILE, which would make your job tremendously easier, but one crucial test checking for similar words in other merged languages' wordlists, has not been implemented yet, and without it, you'll just be asked to check the lists for duplicates. Not to discourage you or anything, but also see my above reply on why the wordlists might never be merged into BIP39.

I don’t have an ETA for this feature, it’s not trivial to implement.

Edit 2: Someone tested our wordlist here:
https://travis-ci.org/github/bitcoin/bips/builds/730941730

The Travis CI build. I don't know what it is, maybe something similar to your tool?

It is not testing your wordlist. Tools such as travis-ci are "hosted continuous integration service" which are automatically called each time there is a new commit (or if it is set, a new pull request) and they run a script (the .yml file) that is sent to their server on each call to run. The script is usually defined to build and run tests but it can be anything.
For example I use it to build my projects on Linux-x64 and also run all my tests there; it helps catch any problems with the build or any bugs in the code that tests can catch.
In case of bips they are running this script that performs some basic checks.

That integration test has nothing to do with wordlists, it is running:

- scripts/link-format-chk.sh
and
- scripts/buildtable.pl

to assemble all the BIPs together in the README file. That makes sense because all the pull requests are for updating BIPs.

[TryNinja]

Our list just got merged! 🎉

https://github.com/bitcoin/bips/commit/cf0b529e78860fa2d4fe77944091aa98c5e04624
https://github.com/bitcoin/bips/blob/master/bip-0039/bip-0039-wordlists.md#portuguese

Thank you for all the help,
@NotATether
@Coding Enthusiast
@ETFbitcoin

And the people who helped write it,
@alegotardo
@bitmover
@brenorb
@kuthullu
@sabotag3x
@Trimegistus

Obrigado!

[Coding Enthusiast]

Our list just got merged! 🎉

I just added the new wordlist to Bitcoin.Net which means next version of my other project FinderOuter will also be able to automatically recover any mnemonics that uses Portuguese word list.

[NotATether]

Do you think that this should get merged into Electrum's seed version system too?

They seem to already have a Portuguese wordlist which is not going to be removed for compatibility reasons. What is the likelihood of such a pull request getting merged? I'm not sure if they are willing to add a "Portuguese v2" wordlist to the codebase given that they are also missing some other official BIP39 wordlists.

[alegotardo]

Our list just got merged! 🎉

Congratulations to everyone who contributed to this !!!
After years without new additions to BIP39, I thought that day would never come.

Do you think that this should get merged into Electrum's seed version system too?

They seem to already have a Portuguese wordlist which is not going to be removed for compatibility reasons. What is the likelihood of such a pull request getting merged? I'm not sure if they are willing to add a "Portuguese v2" wordlist to the codebase given that they are also missing some other official BIP39 wordlists.

Sorry for my ignorance.
But could you tell me where I can find this wordlist?

I changed the language of my electrum desktop wallet to Portuguese and tried to create a new wallet, even so the seeds I received were all in English.

[bitmover]

Sorry for my ignorance.
But could you tell me where I can find this wordlist?

I changed the language of my electrum desktop wallet to Portuguese and tried to create a new wallet, even so the seeds I received were all in English.

I made a simple search and found that:
https://github.com/spesmilo/electrum/blob/master/electrum/wordlist/portuguese.txt

I didn't know that it existed either. This is so crazy.

Do you think that this should get merged into Electrum's seed version system too?

They seem to already have a Portuguese wordlist which is not going to be removed for compatibility reasons. What is the likelihood of such a pull request getting merged? I'm not sure if they are willing to add a "Portuguese v2" wordlist to the codebase given that they are also missing some other official BIP39 wordlists.

It was done in 2014 and it is a Monero list?

Copyright (c) 2014, The Monero Project

It has  only 1626 words and it cannot be used in BIP39 lists like ours.