Evil-Knievel (OP)
Legendary
Offline
Activity: 1260
Merit: 1168
|
|
December 05, 2016, 10:12:29 AM |
|
So far, with the dummy seed: Errors: loracle: 6.15 verynewmember: 5.73 my own submission (post #13): 4.95 Seriously, my submission is shit! I dont wanna win this! He again my shitty test case from from a few days ago: tester.pyfrom retarget import RetargetTest
retarget = RetargetTest() max_error = 0 retarget.seeder("BLOCK_HASH_OF_FIRST_BLOCK_AFTER_CONTEST_GOES HERE") for i in range(9): print "Executing run",i retarget.reset_chain() retarget.randomize_params() retarget.generate_chain() err = retarget.get_total_error() retarget.plot(i) print "Run",i,"-","total error =",err if max_error<err: max_error=err print "Largest error:",max_error retarget.pyimport datetime import random import numpy as np import matplotlib.pyplot as plt
# sudo apt-get install python-tk # pip2 install numpy matplotlib
class RetargetTest(): # experiment with the number of work packages def __init__(self): self.current_time = datetime.datetime.now() self.works_to_create = 3 self.generate_blocks = 100 self.current_height = 0 self.blockchain = [] self.work_packages = [] self.base_target = 0x000000ffffffffffffffffffffffffff self.stretch_number_pows = True self.do_not_randomize_block_times_but_do_always_60_sec = True self.new_miner_every_xth_second = 10 self.how_many_miners_come_or_go = 70242 self.initial_miners = 50 self.miners_kick_in_at_block=50 self.miners_drop_at = 0 self.miners_drop_for=20 self.jitter_size = 0
def seeder(self, hasher): random.seed(hasher)
def randomize_params(self): self.generate_blocks = random.randint(80,500) self.new_miner_every_xth_second = random.randint(5,50) self.how_many_miners_come_or_go = random.randint(0,1000000) self.initial_miners = random.randint(0,1000000) self.miners_kick_in_at_block = random.randint(0,40) self.jitter_size = random.randint(1,7) self.miners_drop_at = random.randint(self.generate_blocks/2,self.generate_blocks) pass
def create_block(self,timestamp, num_pow): return {'time_stamp' : timestamp, 'num_pow' : num_pow, 'first_work_factor':0}
def create_work(self,idx, factor, target): return {'id': idx, 'base_executions_per_second' : factor, 'target' : target}
def addSecs(self,tm, secs): fulldate = tm + datetime.timedelta(seconds=secs) return fulldate
def randomDuration(self): if self.do_not_randomize_block_times_but_do_always_60_sec: return 60 else: return int(random.uniform(25, 120))
def currently_active_miners(self,current_height): if self.current_height<self.miners_kick_in_at_block: return 0
if self.current_height>=self.miners_drop_at and self.current_height<=self.miners_drop_at+self.miners_drop_for: return 0 # get the current active number of miners in relation of blockchain height, # but the number of miners increases by 1 every 10 blocks increases = int(self.current_height/self.new_miner_every_xth_second) * self.how_many_miners_come_or_go return self.initial_miners+increases
def miner_pows_based_on_target(self,work, height, dur): current_target = work["target"] factor = (current_target / self.base_target) * 1.0*dur/60.0 actual_pow_mined = work["base_executions_per_second"] # random jitter actual_pow_mined = abs((actual_pow_mined - self.jitter_size) + random.uniform(self.jitter_size,2*self.jitter_size)) * self.currently_active_miners(height) if actual_pow_mined < 0: actual_pow_mined = 0 actual_pow_mined = actual_pow_mined *factor # rate limit to 20 pows per block if actual_pow_mined > 20: actual_pow_mined = 20 if actual_pow_mined < 0: actual_pow_mined = 0 return actual_pow_mined def kimoto(self,x): return 1 + (0.7084 * pow(((x)/(144)), -1.228)); def retarget_work(self,block, x): targetI = x["target"] pastMass = 0 account_for_block_max = 10 seconds_passed = 0 totalMass = 0 counter = 0 current_block = block current_block_timestamp = self.blockchain[current_block]["time_stamp"]
massive_retarget = False deviation_too_high = False last_two_deviation = 0.0
while True: counter = counter + 1 curmass = self.blockchain[current_block]["num_pow"][x["id"]] pastMass += curmass
# if the maximum block-tx-limit of 20 was reached, do massive retargeting if counter == 1 and pastMass == 20: massive_retarget = True break
# Also if deviation of last two block was too high, do some "magic" if counter == 1 and curmass > 0: last_two_deviation = curmass / 10 if last_two_deviation > 1.25 or last_two_deviation < -0.75: #deviation of over 25% is bad print "last two deviation",last_two_deviation,"at block",block deviation_too_high = True break
for y in self.blockchain[current_block]["num_pow"]: totalMass += self.blockchain[current_block]["num_pow"][y] seconds_passed = (current_block_timestamp - self.blockchain[current_block-1]["time_stamp"]).seconds current_block = current_block - 1 if current_block < 1 or seconds_passed >= 60: # retarget every 120 seconds ~ 1 block on average break
factor = 1 if massive_retarget == True: factor = 0.4 # lower to just 40% elif deviation_too_high == True: factor = 1/last_two_deviation else: if seconds_passed < 1: seconds_passed = 1
pows_per_360_seconds = ((pastMass * 360.0) / seconds_passed) if pows_per_360_seconds>0 and pows_per_360_seconds<1: pows_per_360_seconds = 1
factor = 1 if pows_per_360_seconds > 0: factor = 10*6.0/pows_per_360_seconds if factor<0.9: factor = 0.9 if factor>1.1: factor=1.1 elif pows_per_360_seconds == 0 and totalMass == 0: factor = 1.05 else: factor = 1
#print "seconds",seconds_passed,"blocks",counter,"actual pows",pastMass,"per 360s:",pows_per_360_seconds,"wanted:",60,"factor",factor
targetI = targetI * factor if targetI>self.base_target: targetI = self.base_target if x["id"]==0: self.blockchain[block]["first_work_factor"] = factor x["target"] = targetI
def retarget_works(self,block): for x in self.work_packages: self.retarget_work(block,x)
def reset_chain(self): self.blockchain = [] self.work_packages = [] self.current_height = 0 self.current_time = datetime.datetime.now()
# Here we create up to three different work objects if self.works_to_create>=1: self.work_packages.append(self.create_work(0, 20, self.base_target)) if self.works_to_create>=2: self.work_packages.append(self.create_work(1, 60, self.base_target)) if self.works_to_create>=3: self.work_packages.append(self.create_work(2, 35, self.base_target))
def generate_chain(self): while self.current_height < self.generate_blocks: if self.current_height%1000==0: print " -> generated block",self.current_height dur = self.randomDuration() self.current_time = self.addSecs(self.current_time,dur) # random block generation time block_pows = {} for x in self.work_packages: num_pow = self.miner_pows_based_on_target(x, self.current_height, dur) # mine some POW depending on the current difficulty block_pows[x["id"]] = num_pow self.blockchain.append(self.create_block(self.current_time, block_pows)) self.retarget_works(self.current_height) # This retargeting method is the "critical part here" self.current_height = self.current_height + 1
def get_total_error(self): values = [] ideal = [] for idx in range(len(self.blockchain)): if idx == 0: continue x = self.blockchain[idx] x_minus_one = self.blockchain[idx-1] time_passed = (x["time_stamp"] - x_minus_one["time_stamp"]).seconds strech_normalizer = time_passed / 60.0 sum_x = 0 for y in x["num_pow"]: sum_x += x["num_pow"][y]
if sum_x == 0: ideal.append(0) else: if self.stretch_number_pows == False: ideal.append(self.works_to_create*10*strech_normalizer) else: ideal.append(self.works_to_create*10)
if self.stretch_number_pows == False: values.append(sum_x) else: values.append(sum_x/strech_normalizer) #print values #print ideal total_error = 0 for x in range(len(values)): soll = ideal[x] ist = values[x] total_error += abs(soll-ist) return total_error/len(values)
def plot(self,run): values = [] target_factors = [] ideal = [] for idx in range(len(self.blockchain)): if idx == 0: continue x = self.blockchain[idx] x_minus_one = self.blockchain[idx-1] time_passed = (x["time_stamp"] - x_minus_one["time_stamp"]).seconds strech_normalizer = time_passed / 60.0 sum_x = 0 for y in x["num_pow"]: sum_x += x["num_pow"][y]
if sum_x == 0: ideal.append(0) else: if self.stretch_number_pows == False: ideal.append(self.works_to_create*10*strech_normalizer) else: ideal.append(self.works_to_create*10)
if self.stretch_number_pows == False: values.append(sum_x) else: values.append(sum_x/strech_normalizer) x = range(self.generate_blocks)[1:] y = values
#print "LEN: x",len(x),"y",len(y),"ideal",len(ideal)
#fig = plt.figure() ax0 = plt.subplot(211) if self.stretch_number_pows: ax0.set_ylabel('POW rate per 60s', color='b') else: ax0.set_ylabel('POWs per Block', color='b') ax0.set_xlabel('Block height') ax0.plot(x,y,'-o',x,ideal,'r--') values = [] ideal = [] target_factors = [] for idx in range(len(self.blockchain)): if idx == 0: continue x = self.blockchain[idx] x_minus_one = self.blockchain[idx-1] time_passed = (x["time_stamp"] - x_minus_one["time_stamp"]).seconds strech_normalizer = time_passed / 60.0 sum_x = 0 sum_x += x["num_pow"][0]
if sum_x == 0: ideal.append(0) else: if self.stretch_number_pows == False: ideal.append(10*strech_normalizer) else: ideal.append(10)
#print "sumx",sum_x if self.stretch_number_pows == False: values.append(sum_x) else: values.append(sum_x/strech_normalizer) x = range(self.generate_blocks)[1:] y = values plt.title('All Works: Total POWs') #print "LEN: x",len(x),"y",len(y),"ideal",len(ideal)
ax1 = plt.subplot(212) ax1.plot(x,y,'-o',x,ideal,'r--') ax1.set_xlabel('Block Height') # Make the y-axis label and tick labels match the line color. if self.stretch_number_pows: ax1.set_ylabel('POW rate per 60s', color='b') else: ax1.set_ylabel('POWs per Block', color='b')
for tl in ax1.get_yticklabels(): tl.set_color('b')
ax2 = ax1.twinx() ax2.set_ylim(0.4, 1.6) ax2.bar(x,[x["first_work_factor"] for x in self.blockchain][1:],0.45,color='#deb0b0', alpha=0.2) ax2.set_ylabel('Retargeting Factor', color='r') for tl in ax2.get_yticklabels(): tl.set_color('r') plt.title('First Work: POWs + Retargeting Factor')
plt.savefig('render-' + str(run) + '.png') plt.close()
|
|
|
|
Evil-Knievel (OP)
Legendary
Offline
Activity: 1260
Merit: 1168
|
|
December 05, 2016, 10:18:27 AM |
|
And the plots for the first run of the testbed: loracle: verynewmember: myself:
|
|
|
|
Evil-Knievel (OP)
Legendary
Offline
Activity: 1260
Merit: 1168
|
|
December 05, 2016, 10:46:15 AM |
|
Last minute changes to the testbed are welcome ;-) If you know how to test an even more extreme side case (which an be expected in reality) ... go ahead.
|
|
|
|
Limx Dev
Copper Member
Legendary
Offline
Activity: 2324
Merit: 1348
|
|
December 05, 2016, 11:29:17 AM Last edit: December 05, 2016, 11:39:21 AM by Limx Dev |
|
Last minute changes to the testbed are welcome ;-) If you know how to test an even more extreme side case (which an be expected in reality) ... go ahead.
Try here 72 def kimoto(self,x): return 1 + (0.7084 * pow(((x)/( 28)), -1.228)); And i see the DK3 elemtenst works with 360 sec...that is for a 6 min chain.
|
Bitcore BTX - a UTXO fork of Bitcoin - since 2017
|
|
|
Evil-Knievel (OP)
Legendary
Offline
Activity: 1260
Merit: 1168
|
|
December 05, 2016, 11:33:48 AM |
|
Last minute changes to the testbed are welcome ;-) If you know how to test an even more extreme side case (which an be expected in reality) ... go ahead.
Try here 72 def kimoto(self,x): return 1 + (0.7084 * pow(((x)/(28)), -1.228)); And i see the DK3 elemtenst works with 360 sec...that is for a 6 min chain. Technically we aim for a "6 second" chain here, while in fact it's not a chain! But we want on average every 6 sec / 1 POW to end up with 10 per block ~ 1min. @LIMX Dev: Your suggestion reaches an error of 5.05! Looks close so far ;-) Kimoto28: import datetime import random import numpy as np import matplotlib.pyplot as plt
# sudo apt-get install python-tk # pip2 install numpy matplotlib
class RetargetTest(): # experiment with the number of work packages def __init__(self): self.current_time = datetime.datetime.now() self.works_to_create = 3 self.generate_blocks = 100 self.current_height = 0 self.blockchain = [] self.work_packages = [] self.base_target = 0x000000ffffffffffffffffffffffffff self.stretch_number_pows = True self.do_not_randomize_block_times_but_do_always_60_sec = True self.new_miner_every_xth_second = 10 self.how_many_miners_come_or_go = 70242 self.initial_miners = 50 self.miners_kick_in_at_block=50 self.miners_drop_at = 0 self.miners_drop_for=20 self.jitter_size = 0
def seeder(self, hasher): random.seed(hasher)
def randomize_params(self): self.generate_blocks = random.randint(80,500) self.new_miner_every_xth_second = random.randint(5,50) self.how_many_miners_come_or_go = random.randint(0,1000000) self.initial_miners = random.randint(0,1000000) self.miners_kick_in_at_block = random.randint(0,40) self.jitter_size = random.randint(1,7) self.miners_drop_at = random.randint(self.generate_blocks/2,self.generate_blocks) pass
def create_block(self,timestamp, num_pow): return {'time_stamp' : timestamp, 'num_pow' : num_pow, 'first_work_factor':0}
def create_work(self,idx, factor, target): return {'id': idx, 'base_executions_per_second' : factor, 'target' : target}
def addSecs(self,tm, secs): fulldate = tm + datetime.timedelta(seconds=secs) return fulldate
def randomDuration(self): if self.do_not_randomize_block_times_but_do_always_60_sec: return 60 else: return int(random.uniform(25, 120))
def currently_active_miners(self,current_height): if self.current_height<self.miners_kick_in_at_block: return 0
if self.current_height>=self.miners_drop_at and self.current_height<=self.miners_drop_at+self.miners_drop_for: return 0 # get the current active number of miners in relation of blockchain height, # but the number of miners increases by 1 every 10 blocks increases = int(self.current_height/self.new_miner_every_xth_second) * self.how_many_miners_come_or_go return self.initial_miners+increases
def miner_pows_based_on_target(self,work, height, dur): current_target = work["target"] factor = (current_target / self.base_target) * 1.0*dur/60.0 actual_pow_mined = work["base_executions_per_second"] # random jitter actual_pow_mined = abs((actual_pow_mined - self.jitter_size) + random.uniform(self.jitter_size,2*self.jitter_size)) * self.currently_active_miners(height) if actual_pow_mined < 0: actual_pow_mined = 0 actual_pow_mined = actual_pow_mined *factor # rate limit to 20 pows per block if actual_pow_mined > 20: actual_pow_mined = 20 if actual_pow_mined < 0: actual_pow_mined = 0 return actual_pow_mined def kimoto(self,x): return 1 + (0.7084 * pow(((x)/(28)), -1.228)); def retarget_work(self,block, x): targetI = x["target"] pastMass = 0 counter = 0 current_block = block current_block_timestamp = self.blockchain[current_block]["time_stamp"] adjustment = 0 isFull = True fullCnt = 0 isEmpty = True max_block_reading = 144 emptyCnt = 0 while isFull or isEmpty: if isFull and self.blockchain[current_block]["num_pow"][x["id"]] == 20: fullCnt += 1 else: isFull = False if isEmpty and self.blockchain[current_block]["num_pow"][x["id"]] == 0: emptyCnt += 1 else: isEmpty = False current_block -= 1 if current_block < 1: break current_block = block while True: counter += 1 pastMass += self.blockchain[current_block]["num_pow"][x["id"]] if current_block_timestamp < self.blockchain[current_block-1]["time_stamp"]: current_block_timestamp = self.blockchain[current_block-1]["time_stamp"] seconds_passed = (current_block_timestamp - self.blockchain[current_block-1]["time_stamp"]).seconds current_block -= 1 if seconds_passed < 1: seconds_passed = 1 trs_per_second = float(pastMass) / float(seconds_passed) target_per_second = 10.0 / 60.0 if trs_per_second > 0: adjustment = target_per_second / trs_per_second kim = self.kimoto(pastMass * 30) if adjustment > kim or adjustment < (1.0/kim): break else: adjustment = 1 if current_block < 1 or counter == max_block_reading: break
if fullCnt > 1: adjustment = adjustment / (1 << fullCnt) if emptyCnt > 1: adjustment = adjustment * (1 << emptyCnt) targetI = targetI * adjustment if targetI>self.base_target: targetI = self.base_target if x["id"] == 0: self.blockchain[block]["first_work_factor"] = adjustment x["target"] = targetI #print "Retarget using",counter,"blocks","fullcnt",fullCnt,"emptyCnt",emptyCnt
def retarget_works(self,block): for x in self.work_packages: self.retarget_work(block,x)
def reset_chain(self): self.blockchain = [] self.work_packages = [] self.current_height = 0 self.current_time = datetime.datetime.now()
# Here we create up to three different work objects if self.works_to_create>=1: self.work_packages.append(self.create_work(0, 20, self.base_target)) if self.works_to_create>=2: self.work_packages.append(self.create_work(1, 60, self.base_target)) if self.works_to_create>=3: self.work_packages.append(self.create_work(2, 35, self.base_target))
def generate_chain(self): while self.current_height < self.generate_blocks: if self.current_height%1000==0: print " -> generated block",self.current_height dur = self.randomDuration() self.current_time = self.addSecs(self.current_time,dur) # random block generation time block_pows = {} for x in self.work_packages: num_pow = self.miner_pows_based_on_target(x, self.current_height, dur) # mine some POW depending on the current difficulty block_pows[x["id"]] = num_pow self.blockchain.append(self.create_block(self.current_time, block_pows)) self.retarget_works(self.current_height) # This retargeting method is the "critical part here" self.current_height = self.current_height + 1
def get_total_error(self): values = [] ideal = [] for idx in range(len(self.blockchain)): if idx == 0: continue x = self.blockchain[idx] x_minus_one = self.blockchain[idx-1] time_passed = (x["time_stamp"] - x_minus_one["time_stamp"]).seconds strech_normalizer = time_passed / 60.0 sum_x = 0 for y in x["num_pow"]: sum_x += x["num_pow"][y]
if sum_x == 0: ideal.append(0) else: if self.stretch_number_pows == False: ideal.append(self.works_to_create*10*strech_normalizer) else: ideal.append(self.works_to_create*10)
if self.stretch_number_pows == False: values.append(sum_x) else: values.append(sum_x/strech_normalizer) #print values #print ideal total_error = 0 for x in range(len(values)): soll = ideal[x] ist = values[x] total_error += abs(soll-ist) return total_error/len(values)
def plot(self,run): values = [] target_factors = [] ideal = [] for idx in range(len(self.blockchain)): if idx == 0: continue x = self.blockchain[idx] x_minus_one = self.blockchain[idx-1] time_passed = (x["time_stamp"] - x_minus_one["time_stamp"]).seconds strech_normalizer = time_passed / 60.0 sum_x = 0 for y in x["num_pow"]: sum_x += x["num_pow"][y]
if sum_x == 0: ideal.append(0) else: if self.stretch_number_pows == False: ideal.append(self.works_to_create*10*strech_normalizer) else: ideal.append(self.works_to_create*10)
if self.stretch_number_pows == False: values.append(sum_x) else: values.append(sum_x/strech_normalizer) x = range(self.generate_blocks)[1:] y = values
#print "LEN: x",len(x),"y",len(y),"ideal",len(ideal)
#fig = plt.figure() ax0 = plt.subplot(211) if self.stretch_number_pows: ax0.set_ylabel('POW rate per 60s', color='b') else: ax0.set_ylabel('POWs per Block', color='b') ax0.set_xlabel('Block height') ax0.plot(x,y,'-o',x,ideal,'r--') values = [] ideal = [] target_factors = [] for idx in range(len(self.blockchain)): if idx == 0: continue x = self.blockchain[idx] x_minus_one = self.blockchain[idx-1] time_passed = (x["time_stamp"] - x_minus_one["time_stamp"]).seconds strech_normalizer = time_passed / 60.0 sum_x = 0 sum_x += x["num_pow"][0]
if sum_x == 0: ideal.append(0) else: if self.stretch_number_pows == False: ideal.append(10*strech_normalizer) else: ideal.append(10)
#print "sumx",sum_x if self.stretch_number_pows == False: values.append(sum_x) else: values.append(sum_x/strech_normalizer) x = range(self.generate_blocks)[1:] y = values plt.title('All Works: Total POWs') #print "LEN: x",len(x),"y",len(y),"ideal",len(ideal)
ax1 = plt.subplot(212) ax1.plot(x,y,'-o',x,ideal,'r--') ax1.set_xlabel('Block Height') # Make the y-axis label and tick labels match the line color. if self.stretch_number_pows: ax1.set_ylabel('POW rate per 60s', color='b') else: ax1.set_ylabel('POWs per Block', color='b')
for tl in ax1.get_yticklabels(): tl.set_color('b')
ax2 = ax1.twinx() ax2.set_ylim(0.4, 1.6) ax2.bar(x,[x["first_work_factor"] for x in self.blockchain][1:],0.45,color='#deb0b0', alpha=0.2) ax2.set_ylabel('Retargeting Factor', color='r') for tl in ax2.get_yticklabels(): tl.set_color('r') plt.title('First Work: POWs + Retargeting Factor')
plt.savefig('render-' + str(run) + '.png') plt.close()
|
|
|
|
loracle
Newbie
Offline
Activity: 34
Merit: 0
|
|
December 05, 2016, 11:42:49 AM |
|
Actually, we can improve it even more if we use a very big value for kimoto. It is good according to your test because we update the difficulty at every block. import datetime import random import numpy as np import matplotlib.pyplot as plt
# sudo apt-get install python-tk # pip2 install numpy matplotlib
class RetargetTest(): # experiment with the number of work packages def __init__(self): self.current_time = datetime.datetime.now() self.works_to_create = 3 self.generate_blocks = 100 self.current_height = 0 self.blockchain = [] self.work_packages = [] self.base_target = 0x000000ffffffffffffffffffffffffff self.stretch_number_pows = True self.do_not_randomize_block_times_but_do_always_60_sec = True self.new_miner_every_xth_second = 10 self.how_many_miners_come_or_go = 70242 self.initial_miners = 50 self.miners_kick_in_at_block=50 self.miners_drop_at = 0 self.miners_drop_for=20 self.jitter_size = 0
def seeder(self, hasher): random.seed(hasher)
def randomize_params(self): self.generate_blocks = random.randint(80,500) self.new_miner_every_xth_second = random.randint(5,50) self.how_many_miners_come_or_go = random.randint(0,1000000) self.initial_miners = random.randint(0,1000000) self.miners_kick_in_at_block = random.randint(0,40) self.jitter_size = random.randint(1,7) self.miners_drop_at = random.randint(self.generate_blocks/2,self.generate_blocks) pass
def create_block(self,timestamp, num_pow): return {'time_stamp' : timestamp, 'num_pow' : num_pow, 'first_work_factor':0}
def create_work(self,idx, factor, target): return {'id': idx, 'base_executions_per_second' : factor, 'target' : target}
def addSecs(self,tm, secs): fulldate = tm + datetime.timedelta(seconds=secs) return fulldate
def randomDuration(self): if self.do_not_randomize_block_times_but_do_always_60_sec: return 60 else: return int(random.uniform(25, 120))
def currently_active_miners(self,current_height): if self.current_height<self.miners_kick_in_at_block: return 0
if self.current_height>=self.miners_drop_at and self.current_height<=self.miners_drop_at+self.miners_drop_for: return 0 # get the current active number of miners in relation of blockchain height, # but the number of miners increases by 1 every 10 blocks increases = int(self.current_height/self.new_miner_every_xth_second) * self.how_many_miners_come_or_go return self.initial_miners+increases
def miner_pows_based_on_target(self,work, height, dur): current_target = work["target"] factor = (current_target / self.base_target) * 1.0*dur/60.0 actual_pow_mined = work["base_executions_per_second"] # random jitter actual_pow_mined = abs((actual_pow_mined - self.jitter_size) + random.uniform(self.jitter_size,2*self.jitter_size)) * self.currently_active_miners(height) if actual_pow_mined < 0: actual_pow_mined = 0 actual_pow_mined = actual_pow_mined *factor # rate limit to 20 pows per block if actual_pow_mined > 20: actual_pow_mined = 20 if actual_pow_mined < 0: actual_pow_mined = 0 return actual_pow_mined def kimoto(self,x): return 1 + (0.7084 * pow(((x)/(144)), -1.228)); def retarget_work(self,block, x): targetI = x["target"] pastMass = 0 counter = 0 current_block = block current_block_timestamp = self.blockchain[current_block]["time_stamp"] adjustment = 0 isFull = True fullCnt = 0 isEmpty = True max_block_reading = 144 emptyCnt = 0 while isFull or isEmpty: if isFull and self.blockchain[current_block]["num_pow"][x["id"]] == 20: fullCnt += 1 else: isFull = False if isEmpty and self.blockchain[current_block]["num_pow"][x["id"]] == 0: emptyCnt += 1 else: isEmpty = False current_block -= 1 if current_block < 1: break current_block = block while True: counter += 1 pastMass += self.blockchain[current_block]["num_pow"][x["id"]] if current_block_timestamp < self.blockchain[current_block-1]["time_stamp"]: current_block_timestamp = self.blockchain[current_block-1]["time_stamp"] seconds_passed = (current_block_timestamp - self.blockchain[current_block-1]["time_stamp"]).seconds current_block -= 1 if seconds_passed < 1: seconds_passed = 1 trs_per_second = float(pastMass) / float(seconds_passed) target_per_second = 10.0 / 60.0 if trs_per_second > 0: adjustment = target_per_second / trs_per_second kim = self.kimoto(pastMass * 1000) if adjustment > kim or adjustment < (1.0/kim): break else: adjustment = 1 if current_block < 1 or counter == max_block_reading: break
if fullCnt > 1: adjustment = adjustment / (1 << fullCnt) if emptyCnt > 1: adjustment = adjustment * (2 << emptyCnt) targetI = targetI * adjustment if targetI>self.base_target: targetI = self.base_target if x["id"] == 0: self.blockchain[block]["first_work_factor"] = adjustment x["target"] = targetI #print "Retarget using",counter,"blocks","fullcnt",fullCnt,"emptyCnt",emptyCnt
def retarget_works(self,block): for x in self.work_packages: self.retarget_work(block,x)
def reset_chain(self): self.blockchain = [] self.work_packages = [] self.current_height = 0 self.current_time = datetime.datetime.now()
# Here we create up to three different work objects if self.works_to_create>=1: self.work_packages.append(self.create_work(0, 20, self.base_target)) if self.works_to_create>=2: self.work_packages.append(self.create_work(1, 60, self.base_target)) if self.works_to_create>=3: self.work_packages.append(self.create_work(2, 35, self.base_target))
def generate_chain(self): while self.current_height < self.generate_blocks: if self.current_height%1000==0: print " -> generated block",self.current_height dur = self.randomDuration() self.current_time = self.addSecs(self.current_time,dur) # random block generation time block_pows = {} for x in self.work_packages: num_pow = self.miner_pows_based_on_target(x, self.current_height, dur) # mine some POW depending on the current difficulty block_pows[x["id"]] = num_pow self.blockchain.append(self.create_block(self.current_time, block_pows)) self.retarget_works(self.current_height) # This retargeting method is the "critical part here" self.current_height = self.current_height + 1
def get_total_error(self): values = [] ideal = [] for idx in range(len(self.blockchain)): if idx == 0: continue x = self.blockchain[idx] x_minus_one = self.blockchain[idx-1] time_passed = (x["time_stamp"] - x_minus_one["time_stamp"]).seconds strech_normalizer = time_passed / 60.0 sum_x = 0 for y in x["num_pow"]: sum_x += x["num_pow"][y]
if sum_x == 0: ideal.append(0) else: if self.stretch_number_pows == False: ideal.append(self.works_to_create*10*strech_normalizer) else: ideal.append(self.works_to_create*10)
if self.stretch_number_pows == False: values.append(sum_x) else: values.append(sum_x/strech_normalizer) #print values #print ideal total_error = 0 for x in range(len(values)): soll = ideal[x] ist = values[x] total_error += abs(soll-ist) return total_error/len(values)
def plot(self,run): values = [] target_factors = [] ideal = [] for idx in range(len(self.blockchain)): if idx == 0: continue x = self.blockchain[idx] x_minus_one = self.blockchain[idx-1] time_passed = (x["time_stamp"] - x_minus_one["time_stamp"]).seconds strech_normalizer = time_passed / 60.0 sum_x = 0 for y in x["num_pow"]: sum_x += x["num_pow"][y]
if sum_x == 0: ideal.append(0) else: if self.stretch_number_pows == False: ideal.append(self.works_to_create*10*strech_normalizer) else: ideal.append(self.works_to_create*10)
if self.stretch_number_pows == False: values.append(sum_x) else: values.append(sum_x/strech_normalizer) x = range(self.generate_blocks)[1:] y = values
#print "LEN: x",len(x),"y",len(y),"ideal",len(ideal)
#fig = plt.figure() ax0 = plt.subplot(211) if self.stretch_number_pows: ax0.set_ylabel('POW rate per 60s', color='b') else: ax0.set_ylabel('POWs per Block', color='b') ax0.set_xlabel('Block height') ax0.plot(x,y,'-o',x,ideal,'r--') values = [] ideal = [] target_factors = [] for idx in range(len(self.blockchain)): if idx == 0: continue x = self.blockchain[idx] x_minus_one = self.blockchain[idx-1] time_passed = (x["time_stamp"] - x_minus_one["time_stamp"]).seconds strech_normalizer = time_passed / 60.0 sum_x = 0 sum_x += x["num_pow"][0]
if sum_x == 0: ideal.append(0) else: if self.stretch_number_pows == False: ideal.append(10*strech_normalizer) else: ideal.append(10)
#print "sumx",sum_x if self.stretch_number_pows == False: values.append(sum_x) else: values.append(sum_x/strech_normalizer) x = range(self.generate_blocks)[1:] y = values plt.title('All Works: Total POWs') #print "LEN: x",len(x),"y",len(y),"ideal",len(ideal)
ax1 = plt.subplot(212) ax1.plot(x,y,'-o',x,ideal,'r--') ax1.set_xlabel('Block Height') # Make the y-axis label and tick labels match the line color. if self.stretch_number_pows: ax1.set_ylabel('POW rate per 60s', color='b') else: ax1.set_ylabel('POWs per Block', color='b')
for tl in ax1.get_yticklabels(): tl.set_color('b')
ax2 = ax1.twinx() ax2.set_ylim(0.4, 1.6) ax2.bar(x,[x["first_work_factor"] for x in self.blockchain][1:],0.45,color='#deb0b0', alpha=0.2) ax2.set_ylabel('Retargeting Factor', color='r') for tl in ax2.get_yticklabels(): tl.set_color('r') plt.title('First Work: POWs + Retargeting Factor')
plt.savefig('render-' + str(run) + '.png') plt.close()
Edit: Now max value is 4.71
|
|
|
|
Limx Dev
Copper Member
Legendary
Offline
Activity: 2324
Merit: 1348
|
|
December 05, 2016, 11:56:57 AM |
|
Last minute changes to the testbed are welcome ;-) If you know how to test an even more extreme side case (which an be expected in reality) ... go ahead.
Try here 72 def kimoto(self,x): return 1 + (0.7084 * pow(((x)/(28)), -1.228)); And i see the DK3 elemtenst works with 360 sec...that is for a 6 min chain. Technically we aim for a "6 second" chain here, while in fact it's not a chain! But we want on average every 6 sec / 1 POW to end up with 10 per block ~ 1min. @LIMX Dev: Your suggestion reaches an error of 5.05! Looks close so far ;-) Kimoto28: import datetime import random import numpy as np import matplotlib.pyplot as plt
# sudo apt-get install python-tk # pip2 install numpy matplotlib
class RetargetTest(): # experiment with the number of work packages def __init__(self): self.current_time = datetime.datetime.now() self.works_to_create = 3 self.generate_blocks = 100 self.current_height = 0 self.blockchain = [] self.work_packages = [] self.base_target = 0x000000ffffffffffffffffffffffffff self.stretch_number_pows = True self.do_not_randomize_block_times_but_do_always_60_sec = True self.new_miner_every_xth_second = 10 self.how_many_miners_come_or_go = 70242 self.initial_miners = 50 self.miners_kick_in_at_block=50 self.miners_drop_at = 0 self.miners_drop_for=20 self.jitter_size = 0
def seeder(self, hasher): random.seed(hasher)
def randomize_params(self): self.generate_blocks = random.randint(80,500) self.new_miner_every_xth_second = random.randint(5,50) self.how_many_miners_come_or_go = random.randint(0,1000000) self.initial_miners = random.randint(0,1000000) self.miners_kick_in_at_block = random.randint(0,40) self.jitter_size = random.randint(1,7) self.miners_drop_at = random.randint(self.generate_blocks/2,self.generate_blocks) pass
def create_block(self,timestamp, num_pow): return {'time_stamp' : timestamp, 'num_pow' : num_pow, 'first_work_factor':0}
def create_work(self,idx, factor, target): return {'id': idx, 'base_executions_per_second' : factor, 'target' : target}
def addSecs(self,tm, secs): fulldate = tm + datetime.timedelta(seconds=secs) return fulldate
def randomDuration(self): if self.do_not_randomize_block_times_but_do_always_60_sec: return 60 else: return int(random.uniform(25, 120))
def currently_active_miners(self,current_height): if self.current_height<self.miners_kick_in_at_block: return 0
if self.current_height>=self.miners_drop_at and self.current_height<=self.miners_drop_at+self.miners_drop_for: return 0 # get the current active number of miners in relation of blockchain height, # but the number of miners increases by 1 every 10 blocks increases = int(self.current_height/self.new_miner_every_xth_second) * self.how_many_miners_come_or_go return self.initial_miners+increases
def miner_pows_based_on_target(self,work, height, dur): current_target = work["target"] factor = (current_target / self.base_target) * 1.0*dur/60.0 actual_pow_mined = work["base_executions_per_second"] # random jitter actual_pow_mined = abs((actual_pow_mined - self.jitter_size) + random.uniform(self.jitter_size,2*self.jitter_size)) * self.currently_active_miners(height) if actual_pow_mined < 0: actual_pow_mined = 0 actual_pow_mined = actual_pow_mined *factor # rate limit to 20 pows per block if actual_pow_mined > 20: actual_pow_mined = 20 if actual_pow_mined < 0: actual_pow_mined = 0 return actual_pow_mined def kimoto(self,x): return 1 + (0.7084 * pow(((x)/(28)), -1.228)); def retarget_work(self,block, x): targetI = x["target"] pastMass = 0 counter = 0 current_block = block current_block_timestamp = self.blockchain[current_block]["time_stamp"] adjustment = 0 isFull = True fullCnt = 0 isEmpty = True max_block_reading = 144 emptyCnt = 0 while isFull or isEmpty: if isFull and self.blockchain[current_block]["num_pow"][x["id"]] == 20: fullCnt += 1 else: isFull = False if isEmpty and self.blockchain[current_block]["num_pow"][x["id"]] == 0: emptyCnt += 1 else: isEmpty = False current_block -= 1 if current_block < 1: break current_block = block while True: counter += 1 pastMass += self.blockchain[current_block]["num_pow"][x["id"]] if current_block_timestamp < self.blockchain[current_block-1]["time_stamp"]: current_block_timestamp = self.blockchain[current_block-1]["time_stamp"] seconds_passed = (current_block_timestamp - self.blockchain[current_block-1]["time_stamp"]).seconds current_block -= 1 if seconds_passed < 1: seconds_passed = 1 trs_per_second = float(pastMass) / float(seconds_passed) target_per_second = 10.0 / 60.0 if trs_per_second > 0: adjustment = target_per_second / trs_per_second kim = self.kimoto(pastMass * 30) if adjustment > kim or adjustment < (1.0/kim): break else: adjustment = 1 if current_block < 1 or counter == max_block_reading: break
if fullCnt > 1: adjustment = adjustment / (1 << fullCnt) if emptyCnt > 1: adjustment = adjustment * (1 << emptyCnt) targetI = targetI * adjustment if targetI>self.base_target: targetI = self.base_target if x["id"] == 0: self.blockchain[block]["first_work_factor"] = adjustment x["target"] = targetI #print "Retarget using",counter,"blocks","fullcnt",fullCnt,"emptyCnt",emptyCnt
def retarget_works(self,block): for x in self.work_packages: self.retarget_work(block,x)
def reset_chain(self): self.blockchain = [] self.work_packages = [] self.current_height = 0 self.current_time = datetime.datetime.now()
# Here we create up to three different work objects if self.works_to_create>=1: self.work_packages.append(self.create_work(0, 20, self.base_target)) if self.works_to_create>=2: self.work_packages.append(self.create_work(1, 60, self.base_target)) if self.works_to_create>=3: self.work_packages.append(self.create_work(2, 35, self.base_target))
def generate_chain(self): while self.current_height < self.generate_blocks: if self.current_height%1000==0: print " -> generated block",self.current_height dur = self.randomDuration() self.current_time = self.addSecs(self.current_time,dur) # random block generation time block_pows = {} for x in self.work_packages: num_pow = self.miner_pows_based_on_target(x, self.current_height, dur) # mine some POW depending on the current difficulty block_pows[x["id"]] = num_pow self.blockchain.append(self.create_block(self.current_time, block_pows)) self.retarget_works(self.current_height) # This retargeting method is the "critical part here" self.current_height = self.current_height + 1
def get_total_error(self): values = [] ideal = [] for idx in range(len(self.blockchain)): if idx == 0: continue x = self.blockchain[idx] x_minus_one = self.blockchain[idx-1] time_passed = (x["time_stamp"] - x_minus_one["time_stamp"]).seconds strech_normalizer = time_passed / 60.0 sum_x = 0 for y in x["num_pow"]: sum_x += x["num_pow"][y]
if sum_x == 0: ideal.append(0) else: if self.stretch_number_pows == False: ideal.append(self.works_to_create*10*strech_normalizer) else: ideal.append(self.works_to_create*10)
if self.stretch_number_pows == False: values.append(sum_x) else: values.append(sum_x/strech_normalizer) #print values #print ideal total_error = 0 for x in range(len(values)): soll = ideal[x] ist = values[x] total_error += abs(soll-ist) return total_error/len(values)
def plot(self,run): values = [] target_factors = [] ideal = [] for idx in range(len(self.blockchain)): if idx == 0: continue x = self.blockchain[idx] x_minus_one = self.blockchain[idx-1] time_passed = (x["time_stamp"] - x_minus_one["time_stamp"]).seconds strech_normalizer = time_passed / 60.0 sum_x = 0 for y in x["num_pow"]: sum_x += x["num_pow"][y]
if sum_x == 0: ideal.append(0) else: if self.stretch_number_pows == False: ideal.append(self.works_to_create*10*strech_normalizer) else: ideal.append(self.works_to_create*10)
if self.stretch_number_pows == False: values.append(sum_x) else: values.append(sum_x/strech_normalizer) x = range(self.generate_blocks)[1:] y = values
#print "LEN: x",len(x),"y",len(y),"ideal",len(ideal)
#fig = plt.figure() ax0 = plt.subplot(211) if self.stretch_number_pows: ax0.set_ylabel('POW rate per 60s', color='b') else: ax0.set_ylabel('POWs per Block', color='b') ax0.set_xlabel('Block height') ax0.plot(x,y,'-o',x,ideal,'r--') values = [] ideal = [] target_factors = [] for idx in range(len(self.blockchain)): if idx == 0: continue x = self.blockchain[idx] x_minus_one = self.blockchain[idx-1] time_passed = (x["time_stamp"] - x_minus_one["time_stamp"]).seconds strech_normalizer = time_passed / 60.0 sum_x = 0 sum_x += x["num_pow"][0]
if sum_x == 0: ideal.append(0) else: if self.stretch_number_pows == False: ideal.append(10*strech_normalizer) else: ideal.append(10)
#print "sumx",sum_x if self.stretch_number_pows == False: values.append(sum_x) else: values.append(sum_x/strech_normalizer) x = range(self.generate_blocks)[1:] y = values plt.title('All Works: Total POWs') #print "LEN: x",len(x),"y",len(y),"ideal",len(ideal)
ax1 = plt.subplot(212) ax1.plot(x,y,'-o',x,ideal,'r--') ax1.set_xlabel('Block Height') # Make the y-axis label and tick labels match the line color. if self.stretch_number_pows: ax1.set_ylabel('POW rate per 60s', color='b') else: ax1.set_ylabel('POWs per Block', color='b')
for tl in ax1.get_yticklabels(): tl.set_color('b')
ax2 = ax1.twinx() ax2.set_ylim(0.4, 1.6) ax2.bar(x,[x["first_work_factor"] for x in self.blockchain][1:],0.45,color='#deb0b0', alpha=0.2) ax2.set_ylabel('Retargeting Factor', color='r') for tl in ax2.get_yticklabels(): tl.set_color('r') plt.title('First Work: POWs + Retargeting Factor')
plt.savefig('render-' + str(run) + '.png') plt.close()
Can you test here 72 return 1 + (0.7084 * pow(((x)/(72)), -1.228)) and her max_block_reading = 72
|
Bitcore BTX - a UTXO fork of Bitcoin - since 2017
|
|
|
Evil-Knievel (OP)
Legendary
Offline
Activity: 1260
Merit: 1168
|
|
December 05, 2016, 11:59:58 AM |
|
Can you test here 72 return 1 + (0.7084 * pow(((x)/(72)), -1.228)) and her max_block_reading = 72
Worse: Executing run 0 -> generated block 0 Run 0 - total error = 2.17624878297 Executing run 1 -> generated block 0 Run 1 - total error = 2.23758974444 Executing run 2 -> generated block 0 Run 2 - total error = 4.59423461528 Executing run 3 -> generated block 0 Run 3 - total error = 2.13096649602 Executing run 4 -> generated block 0 Run 4 - total error = 5.36702805093 Executing run 5 -> generated block 0 Run 5 - total error = 1.86506009398 Executing run 6 -> generated block 0 Run 6 - total error = 1.93048505133 Executing run 7 -> generated block 0 Run 7 - total error = 1.78515230057 Executing run 8 -> generated block 0 Run 8 - total error = 2.10473265039 Largest error: 5.36702805093
|
|
|
|
Limx Dev
Copper Member
Legendary
Offline
Activity: 2324
Merit: 1348
|
|
December 05, 2016, 12:07:44 PM |
|
Can you test here 72 return 1 + (0.7084 * pow(((x)/(72)), -1.228)) and her max_block_reading = 72
Worse: Executing run 0 -> generated block 0 Run 0 - total error = 2.17624878297 Executing run 1 -> generated block 0 Run 1 - total error = 2.23758974444 Executing run 2 -> generated block 0 Run 2 - total error = 4.59423461528 Executing run 3 -> generated block 0 Run 3 - total error = 2.13096649602 Executing run 4 -> generated block 0 Run 4 - total error = 5.36702805093 Executing run 5 -> generated block 0 Run 5 - total error = 1.86506009398 Executing run 6 -> generated block 0 Run 6 - total error = 1.93048505133 Executing run 7 -> generated block 0 Run 7 - total error = 1.78515230057 Executing run 8 -> generated block 0 Run 8 - total error = 2.10473265039 Largest error: 5.36702805093
I think a harder retarget is better. ;-)
|
Bitcore BTX - a UTXO fork of Bitcoin - since 2017
|
|
|
Evil-Knievel (OP)
Legendary
Offline
Activity: 1260
Merit: 1168
|
|
December 05, 2016, 12:33:20 PM |
|
For some odd reason, kimoto underperforms under realistic conditions. Maybe we have to make our tests a bit more realistic, but there is not much time left. For large block-time-variations the kimoto approach tends to error significantly.
|
|
|
|
Limx Dev
Copper Member
Legendary
Offline
Activity: 2324
Merit: 1348
|
|
December 05, 2016, 12:54:53 PM |
|
For some odd reason, kimoto underperforms under realistic conditions. Maybe we have to make our tests a bit more realistic, but there is not much time left. For large block-time-variations the kimoto approach tends to error significantly. Yes that is true, i recommend make the diffretarget not to soft. Here a bad Example.
|
Bitcore BTX - a UTXO fork of Bitcoin - since 2017
|
|
|
HunterMinerCrafter
|
|
December 05, 2016, 03:23:48 PM |
|
Yah, this testbench is not so great. Using a variant of EK's alg, I reached a largest error of 2.52601249956 on the reference seed. Without cheating, this time. Is that good? I think I still have room for improvement, but I am getting tired. EK: Can I submit my entry by pm?
|
|
|
|
Evil-Knievel (OP)
Legendary
Offline
Activity: 1260
Merit: 1168
|
|
December 05, 2016, 06:28:25 PM |
|
Yah, this testbench is not so great. Using a variant of EK's alg, I reached a largest error of 2.52601249956 on the reference seed. Without cheating, this time. Is that good? I think I still have room for improvement, but I am getting tired. EK: Can I submit my entry by pm? Of course ;-)
|
|
|
|
HunterMinerCrafter
|
|
December 05, 2016, 06:54:24 PM |
|
Of course ;-)
NM, here is the best I've gotten to so far. We'll call it a study in realistic simulation being hard and overfitting being too easy. Somewhat ironically, it is your original algorithm with different parameters and basically only one (subtle) change. http://pastebin.com/YD6bYzyPThis beats "similar optimization approaches" for my few attempts at a hand-rolled re-target, but I could probably score better with some more time spent there. I'm curious to see if anyone can beat it, as well. Let me know what you think.
|
|
|
|
Evil-Knievel (OP)
Legendary
Offline
Activity: 1260
Merit: 1168
|
|
December 05, 2016, 07:18:07 PM |
|
Let me know what you think. I think a) that it is sweet to see that my first "idea" was not so bad after all and b) that Largest error: 1.94385646144 seems hard to beat and last but not least c) that I hope to get the one or another improvement suggestion from you, even after the contest ist over and we have seen this method "in action" in the upcoming 0.9.0 testnet
|
|
|
|
loracle
Newbie
Offline
Activity: 34
Merit: 0
|
|
December 05, 2016, 07:24:32 PM |
|
Does the contest end at 9:24 or at 7:51 ?
|
|
|
|
Evil-Knievel (OP)
Legendary
Offline
Activity: 1260
Merit: 1168
|
|
December 05, 2016, 07:33:23 PM |
|
Does the contest end at 9:24 or at 7:51 ?
7:51 forum time! In about 20 minutes. And it's about elaborating on different ways, just we had it here until now ;-) So I am very eager to see how the different approaches will eventually perform. Resubmissions of already present solutions with minor changes should be not allowed unless it's the own ones tweaked. I will try your PM submission now, loracle ;-) Thanks so far!
|
|
|
|
Limx Dev
Copper Member
Legendary
Offline
Activity: 2324
Merit: 1348
|
|
December 05, 2016, 07:34:24 PM |
|
Let me know what you think. I think a) that it is sweet to see that my first "idea" was not so bad after all and b) that Largest error: 1.94385646144 seems hard to beat and last but not least c) that I hope to get the one or another improvement suggestion from you, even after the contest ist over and we have seen this method "in action" in the upcoming 0.9.0 testnet Here can you see a active DK3. Yes the retarget is hard but works fine. It is a 5 min Chain. This is with 72 and blockreading count...i think over 400 blocks are possible https://chainz.cryptoid.info/erc/ -> Last 100 -> Diff
|
Bitcore BTX - a UTXO fork of Bitcoin - since 2017
|
|
|
Evil-Knievel (OP)
Legendary
Offline
Activity: 1260
Merit: 1168
|
|
December 05, 2016, 07:58:50 PM |
|
The random seed for testing is, as described, the following string (with leading 0x) 0x00000000000000000065e874f49c4ab116f5a7a504a527f0088a3e86d2bda439 I will start the evaluation now!
|
|
|
|
Evil-Knievel (OP)
Legendary
Offline
Activity: 1260
Merit: 1168
|
|
December 05, 2016, 08:10:26 PM |
|
Hunterminercrafter:Largest error: 1.89787520814 L'oracle (PM submission from 07:42:50 PM, someone can verify this in my account if wanted):Largest error: 1.89720766492 My own one (shame on me)Largest error: 5.31689202348 VeryNewMember:Largest error: 5.9709180146 Here, all algorithms in zipped form. Change the first line in tester.py to evaluate a specific submission: http://www.xup.in/dl,17161517/retarget_algorithms.tar.gz/Does that seem correct so far? Unbelieveable, how close the two best submissions were. If everyone agrees with the result, I would need loracle's BTC address to get rid of my hard earned BTC.
|
|
|
|
|