SMP, 64bit Unix and Kernel Compilation
Guntis BarzdinsGirts Folkmanis
K palaist MPI[guntisb@zars mpi]$ ls -ltotal 392-rw-rw-r-- 1 guntisb guntisb 122 Apr 28 07:08 Makefile-rw-rw-r-- 1 guntisb guntisb 13 May 17 14:33 mfile-rw-rw-r-- 1 guntisb guntisb 344 May 12 09:28 mpi.jdl-rw-rw-r-- 1 guntisb guntisb 2508 Apr 28 07:08 mpi.sh-rwxrwxr-x 1 guntisb guntisb 331899 May 17 14:48 passtonext-rw-rw-r-- 1 guntisb guntisb 3408 Apr 28 07:08 passtonext.c-rw-rw-r-- 1 guntisb guntisb 2132 May 17 14:48 passtonext.o[guntisb@zars mpi]$ more mfilelocalhost:4
[guntisb@zars mpi]$[guntisb@zars mpi]$ makempicc passtonext.c -o passtonext -lmpich -lm[guntisb@zars mpi]$ mpirun -np 2 -machinefile mfile passtonextguntisb@localhost's password:Nodename=zars.latnet.lv Rank=0 Size=2INFO: zars.latnet.lv (0 of 2) sent 73 value to 1 of 2INFO: zars.latnet.lv (0 of 2) received 74 value from 1 of 2Nodename=zars.latnet.lv Rank=1 Size=2INFO: zars.latnet.lv (1 of 2) received 73 value from 0 of 2INFO: zars.latnet.lv (1 of 2) sent 73+1=74 value to 0 of 2[guntisb@zars mpi]$ Paldies Jnim Tragheimam!!
3b mjas darba variantsJa ir izpildits 3a mjas darbs, tad 3b da var uzrakstt un notestt pats savu MPI programmu, kas rins ar vismaz 4 procesiem. MPI programmu drkst nedarbint Grid!
Message-passing performance comparisonSMP MPI performanceGigabitEthernet MPI performance
Chart1
0.2814757.3713573.0639870.176333
0.6139914.5477865.9856520.392785
0.92692821.1595078.9933650.499961
1.21414428.79959411.8925040.786296
1.83384643.72531817.8275751.274626
2.46317158.0383723.7161581.606568
3.6870285.2211534.2666492.016553
3.96072594.55610236.6271222.438502
4.885092113.81749546.3351842.71544
5.790828137.76145653.0607584.026536
6.413572152.97400959.1526473.350554
7.368076170.28560270.079453.831907
8.203419197.61352375.3156854.55265
8.79215206.89368681.7001455.655495
9.646528228.85791292.3903995.426617
10.644633250.41229498.9083398.508124
13.79629324.127558120.6036099.147833
14.647996350.064451133.3405619.74361
15.435718364.706047122.82228610.552526
18.507878437.192419167.62115713.154808
19.545046462.939828181.35529713.079803
20.402282369.348908171.11352211.692634
27.974254497.147959242.17765119.103481
29.138987572.44007262.04488619.511098
29.968021543.207771257.42705220.192376
36.983623707.531119308.7725729.997626
38.64132760.725658323.31975125.90242
38.328893612.109993319.56274929.035245
54.872792870.859807446.5656941.735663
56.489209961.219338458.96678537.578825
57.595077758.687038435.19020541.791783
75.9039381000.434619534.24871658.094607
73.9393711083.917552541.85102359.023802
74.141053913.813495542.00898462.554444
107.2251671227.421415764.05805967.970935
109.4224611355.946837780.96204787.74085
109.1032071178.534485775.98643673.465665
138.19411466.745324953.502881119.542283
142.6563121404.286583966.562794104.978561
142.341211389.989651941.8648289.494004
202.3576311724.1559481255.968218158.880963
209.525711711.4644591267.68997156.118055
198.0298071624.1463221252.544659134.762982
255.8777441852.7836731492.127754177.606592
267.9571191915.5399471503.721693223.523028
254.3999591764.7446591482.758837228.529378
349.5140142036.4580141770.020829338.73584
371.7808262159.7597951783.600697287.490738
354.7484071975.1681731743.762752327.066776
432.8338192128.3082541968.873759433.921543
474.4960482311.1110551958.28841435.18533
435.208942079.100731945.381387336.127091
554.8894472213.626022207.641599619.964505
618.2340462507.7581132241.552252552.195503
554.2370712191.7474152244.964955582.984516
650.4876092181.7784452377.219095638.214068
729.588772592.9058752389.618441610.301585
650.566222167.0314052358.087989705.164857
765.9324252237.1045892551.238118937.898778
943.00732514.9710382729.443958905.954846
786.4208942246.075452552.757392966.494976
863.5860212263.3207512715.5842041078.395605
1079.3777912498.8959292705.4491911140.150726
859.0415782377.8022662691.7776891125.115603
957.9282142383.4496542835.1465531584.461976
1234.1848442528.6624522809.3028341479.862384
960.1059712375.5531182828.596321484.113874
924.5415982382.3680952884.6070581666.423811
1228.9351812505.5730372926.8699251654.527238
1209.3194432222.7868753007.519671668.604029
1383.5845982203.3100033068.9415112060.832715
1467.8976292908.2160863090.16132060.31493
1426.425272315.6772643062.6003622075.648832
1411.6546092224.351943105.0237012414.289072
1618.0847883086.5931663017.3169612375.29744
1396.1111552374.6092842993.1870862377.499718
1559.086882261.1548843010.5708052418.599859
1629.0289223269.3216933140.9253072513.455461
1504.5789052426.237463137.0899022396.62052
1579.8937422273.3931113138.3684532104.189811
1722.5120243424.914482999.2903892217.107943
1437.5914282467.9654873036.4436532252.584054
1458.6236572315.8918793141.1677972084.208735
1553.5302363512.4810613063.1059972160.070803
1488.3655052490.6695163012.3403092178.419039
1136.6307692337.3150423112.7162261420.054821
1401.5058513541.2801773091.6515941428.255358
1373.5910522536.0024243037.9628351468.932822
1298.3921072395.6589253162.9171571483.917741
1405.7753363459.8289723033.4987171445.590028
1353.6340382593.4647493111.864971478.731663
1234.0712182405.7469213210.4584971469.677393
1386.0168963452.0462533084.962141485.791533
1218.9794852629.5764433155.1917761487.170593
1394.1168032450.4991373251.3063071482.944557
1433.8852383361.2400483170.4554921480.853378
1400.613612670.2955073145.1721751453.466099
1265.734952457.3278873214.8091651494.811133
1439.7992843300.7374813168.2213281496.274828
1296.2451442679.7980013166.7579851496.034623
1451.7242482487.3027553236.435031515.591772
1479.1986763211.7686163176.7876871493.953952
1438.5847892721.8745063135.4162181501.742444
1488.5055382494.7687333252.837781525.985278
1568.8330063213.4976083121.858291525.859675
1306.9560542743.5634093159.1721251569.776899
1379.9223092931.1768333247.5097781539.631626
1404.7885063176.6771083163.1444721550.712877
1381.6956042904.2720843191.868481533.622164
1369.7835662839.5826223184.66811632.316899
1404.002223157.6462493180.108151557.434898
1385.2076312840.9146013156.4815841571.268858
1388.0243192956.5964363221.7604121523.095538
1411.7928213151.5442383205.9580471501.430489
1386.5368832930.1163523224.534791513.396937
1380.8771952977.7796943206.6093681555.372109
1390.1360423169.5327493171.8352771596.992117
1385.5022882971.4928883221.706451556.496431
1375.8831243026.3377413200.6374611490.720961
1405.8311083159.558423219.4764381514.251193
1377.964953044.1174053225.61681511.613287
1380.2121033061.1509753184.3158711496.591076
1412.1232133147.7486063204.7017611497.268366
1348.2443283061.4463433159.4273731508.278525
MPICH25 us
MP_Lite3 us
PVM33 us
LAM/MPI 1 us
MPICH
LAM
MP_Lite
PVM
Message size in Bytes
Throughput in Mbps
np.mini.fe
MPICHLAMMP_LitePVM
10.2814757.3713573.0639870.176333
20.6139914.5477865.9856520.392785
30.92692821.1595078.9933650.499961
41.21414428.79959411.8925040.786296
61.83384643.72531817.8275751.274626
82.46317158.0383723.7161581.606568
123.6870285.2211534.2666492.016553
133.96072594.55610236.6271222.438502
164.885092113.81749546.3351842.71544
195.790828137.76145653.0607584.026536
216.413572152.97400959.1526473.350554
247.368076170.28560270.079453.831907
278.203419197.61352375.3156854.55265
298.79215206.89368681.7001455.655495
329.646528228.85791292.3903995.426617
3510.644633250.41229498.9083398.508124
4513.79629324.127558120.6036099.147833
4814.647996350.064451133.3405619.74361
5115.435718364.706047122.82228610.552526
6118.507878437.192419167.62115713.154808
6419.545046462.939828181.35529713.079803
6720.402282369.348908171.11352211.692634
9327.974254497.147959242.17765119.103481
9629.138987572.44007262.04488619.511098
9929.968021543.207771257.42705220.192376
12536.983623707.531119308.7725729.997626
12838.64132760.725658323.31975125.90242
13138.328893612.109993319.56274929.035245
18954.872792870.859807446.5656941.735663
19256.489209961.219338458.96678537.578825
19557.595077758.687038435.19020541.791783
25375.9039381000.434619534.24871658.094607
25673.9393711083.917552541.85102359.023802
25974.141053913.813495542.00898462.554444
381107.2251671227.421415764.05805967.970935
384109.4224611355.946837780.96204787.74085
387109.1032071178.534485775.98643673.465665
509138.19411466.745324953.502881119.542283
512142.6563121404.286583966.562794104.978561
515142.341211389.989651941.8648289.494004
765202.3576311724.1559481255.968218158.880963
768209.525711711.4644591267.68997156.118055
771198.0298071624.1463221252.544659134.762982
1021255.8777441852.7836731492.127754177.606592
1024267.9571191915.5399471503.721693223.523028
1027254.3999591764.7446591482.758837228.529378
1533349.5140142036.4580141770.020829338.73584
1536371.7808262159.7597951783.600697287.490738
1539354.7484071975.1681731743.762752327.066776
2045432.8338192128.3082541968.873759433.921543
2048474.4960482311.1110551958.28841435.18533
2051435.208942079.100731945.381387336.127091
3069554.8894472213.626022207.641599619.964505
3072618.2340462507.7581132241.552252552.195503
3075554.2370712191.7474152244.964955582.984516
4093650.4876092181.7784452377.219095638.214068
4096729.588772592.9058752389.618441610.301585
4099650.566222167.0314052358.087989705.164857
6141765.9324252237.1045892551.238118937.898778
6144943.00732514.9710382729.443958905.954846
6147786.4208942246.075452552.757392966.494976
8189863.5860212263.3207512715.5842041078.395605
81921079.3777912498.8959292705.4491911140.150726
8195859.0415782377.8022662691.7776891125.115603
12285957.9282142383.4496542835.1465531584.461976
122881234.1848442528.6624522809.3028341479.862384
12291960.1059712375.5531182828.596321484.113874
16381924.5415982382.3680952884.6070581666.423811
163841228.9351812505.5730372926.8699251654.527238
163871209.3194432222.7868753007.519671668.604029
245731383.5845982203.3100033068.9415112060.832715
245761467.8976292908.2160863090.16132060.31493
245791426.425272315.6772643062.6003622075.648832
327651411.6546092224.351943105.0237012414.289072
327681618.0847883086.5931663017.3169612375.29744
327711396.1111552374.6092842993.1870862377.499718
491491559.086882261.1548843010.5708052418.599859
491521629.0289223269.3216933140.9253072513.455461
491551504.5789052426.237463137.0899022396.62052
655331579.8937422273.3931113138.3684532104.189811
655361722.5120243424.914482999.2903892217.107943
655391437.5914282467.9654873036.4436532252.584054
983011458.6236572315.8918793141.1677972084.208735
983041553.5302363512.4810613063.1059972160.070803
983071488.3655052490.6695163012.3403092178.419039
1310691136.6307692337.3150423112.7162261420.054821
1310721401.5058513541.2801773091.6515941428.255358
1310751373.5910522536.0024243037.9628351468.932822
1966051298.3921072395.6589253162.9171571483.917741
1966081405.7753363459.8289723033.4987171445.590028
1966111353.6340382593.4647493111.864971478.731663
2621411234.0712182405.7469213210.4584971469.677393
2621441386.0168963452.0462533084.962141485.791533
2621471218.9794852629.5764433155.1917761487.170593
3932131394.1168032450.4991373251.3063071482.944557
3932161433.8852383361.2400483170.4554921480.853378
3932191400.613612670.2955073145.1721751453.466099
5242851265.734952457.3278873214.8091651494.811133
5242881439.7992843300.7374813168.2213281496.274828
5242911296.2451442679.7980013166.7579851496.034623
7864291451.7242482487.3027553236.435031515.591772
7864321479.1986763211.7686163176.7876871493.953952
7864351438.5847892721.8745063135.4162181501.742444
10485731488.5055382494.7687333252.837781525.985278
10485761568.8330063213.4976083121.858291525.859675
10485791306.9560542743.5634093159.1721251569.776899
15728611379.9223092931.1768333247.5097781539.631626
15728641404.7885063176.6771083163.1444721550.712877
15728671381.6956042904.2720843191.868481533.622164
20971491369.7835662839.5826223184.66811632.316899
20971521404.002223157.6462493180.108151557.434898
20971551385.2076312840.9146013156.4815841571.268858
31457251388.0243192956.5964363221.7604121523.095538
31457281411.7928213151.5442383205.9580471501.430489
31457311386.5368832930.1163523224.534791513.396937
41943011380.8771952977.7796943206.6093681555.372109
41943041390.1360423169.5327493171.8352771596.992117
41943071385.5022882971.4928883221.706451556.496431
62914531375.8831243026.3377413200.6374611490.720961
62914561405.8311083159.558423219.4764381514.251193
62914591377.964953044.1174053225.61681511.613287
83886051380.2121033061.1509753184.3158711496.591076
83886081412.1232133147.7486063204.7017611497.268366
83886111348.2443283061.4463433159.4273731508.278525
np.mini.fe
MPICH25 us
MP_Lite3 us
PVM33 us
LAM/MPI 1 us
MPICH
LAM
MP_Lite
PVM
Message size in Bytes
Throughput in Mbps
Chart1
0.0381610.0381450.0381180.0381320.0381260.038127
0.0763020.0762610.0762670.0762680.0762830.076301
0.1144530.1144140.1144150.1143730.1144250.114405
0.1525440.1527130.152450.1526710.1525650.152617
0.2289820.2287530.2286350.2287520.2289690.228918
0.3053960.305290.3054070.3051840.3052490.305144
0.4581630.4574640.4580360.4577430.4579850.457569
0.4958890.4956890.4961750.495670.4958270.495861
0.6116420.6103650.6106370.6103760.6105160.61029
0.7249960.7250920.7245420.724540.7246820.724713
0.8012740.8010740.8010890.8013380.8013070.801036
0.9154480.9155140.9155440.9154860.915530.915308
1.029591.0299481.0299541.0290441.0298171.029561
1.1080891.1063371.1067961.1064071.1060731.10712
1.2216741.2207491.2201671.2208651.2205061.220066
1.3355461.3345961.3361011.3356121.3358591.334988
1.7178291.7159821.7171491.7167461.7169511.716428
1.8309071.8310431.8315881.8315111.830781.830857
1.9459031.9454451.9449291.9465781.9457791.943578
2.3280232.3270962.3270842.3269772.3265852.330301
2.4413662.4415552.4424812.4413572.4410192.441029
2.5575852.5558972.5568792.5541022.555442.554578
3.5487383.5477343.5465583.5480113.5471013.547311
3.661793.6620323.6588813.659893.6615713.661703
3.7779913.7765063.7765393.776363.7748653.775011
4.7681634.7710534.7686354.7687394.7676354.765248
4.8864644.8831864.8837984.8828524.8800534.885023
4.9968794.9979834.9993034.9971944.9964454.99865
7.2092517.2128127.2151137.1985047.2077757.209027
7.3236257.3214957.3263967.3094587.3230977.321207
7.4424747.4253857.440837.4346687.4420077.428943
9.6522379.6130089.6570019.653359.6512389.622536
9.7648029.7657829.7668819.7611929.7639399.764603
9.8794369.8764959.880429.8759839.8784159.879043
14.53867414.53416814.53998814.337214.53245314.530454
14.65129714.64826214.64839214.64251714.51142312.948124
14.76172914.76257214.76274714.76221614.76055912.799333
19.41578719.35014719.42957817.76957416.62942918.938847
19.53775319.53164519.5237414.19078319.52795614.573324
19.64435519.64647319.64608415.98338819.64248319.633313
29.1912329.04698729.1945821.75095829.14253829.179417
29.28500729.234929.30567216.38225827.52674321.304491
29.41757629.31410229.45549723.78372924.50057329.396421
38.99019838.95085338.97323727.9363938.93946433.92745
39.06016239.06327839.08003835.43551839.05562234.029107
39.17397139.1935439.13162233.33621337.39399928.967598
40.42404541.29397553.02596952.99856542.64145955.766694
58.59628258.59549841.68271739.30369746.2875246.176994
42.55584744.88391334.823830.90072558.60732241.359307
60.33208262.4131156.40987340.11293456.39788378.005301
77.43879860.79363558.09532670.24980665.05398165.253829
70.88820772.12204458.96681264.54488458.53885559.551467
105.31351978.95525277.433515106.92361589.23702875.658416
61.55672980.9819394.68205780.902905110.51769174.053444
100.618714100.843534108.88967172.252514117.27657281.175949
150.52657190.919694116.653497100.196237103.1209881.038003
127.827082104.68865129.619563112.914765142.38791480.156104
145.13887398.204626116.486837120.49974397.48576989.868656
127.383841138.892796128.619426117.590748117.973491121.26169
181.174485127.791055131.301311138.860339176.990375135.233866
129.835391118.53153152.884766150.314208149.432742118.779276
180.634156154.084244203.064639153.702312160.12104154.436174
174.68441153.942764204.330629153.703529161.0131153.736802
227.98184156.797503157.511262154.007173155.925881154.496155
225.636208233.632903226.578122230.726719227.896725231.194076
226.724455232.644153232.228915231.852167230.743621228.647683
226.728076228.594239227.780522229.009944228.378106228.977752
274.027817269.996498278.828066179.231819284.26482151.900991
277.982945276.351733280.036482179.809828298.113839151.71794
283.153327290.893004293.326027167.589783295.438224149.266285
359.192948282.308403217.982385150.260275301.604907138.479871
340.159738281.911633212.810675185.411671302.483101146.782264
353.768143285.559468218.5164166.170952300.548004149.177851
351.665585298.956127218.861579167.213347324.164554185.567085
352.58368298.694194192.81117166.452061331.36703171.02341
352.558225298.940859209.847921166.116185328.275981173.483823
380.88954337.75593227.562565175.649182382.758364158.354173
393.521036339.017825234.7294175.347077383.206287175.782227
399.47588340.433648228.860379205.706406385.353738181.838338
431.790947354.198557258.153917203.928035404.706746172.823034
435.09208354.918884251.96351191.906937404.334847168.731342
431.57609355.824888217.435538191.047828404.319383160.425975
468.765745366.325772242.374732202.360242440.321872141.959444
465.535377366.756995232.581276206.707359439.660667156.283084
466.830917367.14946230.972724210.697712439.836357178.403186
489.291443345.743653245.804565208.269152452.386669187.23302
489.482042347.012124244.922947207.366292454.11793187.256014
489.573917346.881379238.217864204.455973450.701848159.658811
508.635057371.103069244.949404210.622987465.000979158.873301
509.192669369.845791256.20503206.692665467.058584152.366628
510.653796370.128594261.274386218.483467465.63005171.929814
519.250495374.481988250.074219227.347614480.507328193.309226
518.899329378.16968250.487368227.165802479.198842193.938595
519.862082376.559703254.481736222.787036480.306937189.632506
528.946367376.746706269.960148229.009271491.955472172.189377
528.862536377.886078260.652606228.316978492.637716152.419228
528.850547376.280992260.235466231.023518491.916126175.261832
536.336321381.906719264.72252240.330672499.874496193.202924
535.795764383.392169265.781876237.849558500.014481185.231732
535.325936381.979839263.210258241.231965500.467805174.561879
539.890965396.546122260.627791241.693723479.873969164.869483
539.000543394.814811271.640694243.593383480.917512186.811975
539.783632393.624866276.810349242.460141480.066764190.284062
543.273793400.516715267.08372242.294513490.194555167.17891
542.786753400.374608277.078206243.552917491.21092185.975476
543.346043398.107209269.754803237.199879491.630663191.492171
545.049287402.142993270.437248236.900217498.11037174.897829
545.238477402.332566270.114694243.542479498.10248188.466939
545.113937401.885761269.094139247.621159497.817105176.994108
546.336638404.973404274.842188249.018302503.096422187.048445
546.628151404.84732269.798585244.891435503.526966188.930202
546.346304404.582477267.452855242.786406503.356768183.256671
547.66063407.674074268.646931251.142982507.58838190.032713
547.249052408.122113279.05632244.323469507.446331193.622106
547.299578407.603619281.26667242.413983507.485904179.115823
548.556058409.868781276.949382247.657302510.078797191.512844
548.308041409.063322273.889194245.47475509.655568192.755163
548.371515408.070974277.321616248.626628509.817165189.158894
549.383346411.255734277.619735250.95497512.06178188.311532
549.132201411.299219274.050815248.575501512.168575185.084858
549.071005409.721312281.301741248.521657511.807629181.936362
549.587164411.011102276.236099246.914032513.202478186.431618
549.75903411.306707276.982361249.116277512.940591186.61904
549.516123409.421407277.877982247.577508512.938407183.386672
MPICH
MP_Lite
raw TCP
PVM
LAM/MPI
MPI/Pro
raw TCP
MPICH
LAM-MPI
MPI/Pro
MP_Lite
PVM
Message size in Bytes
Throughput in Mbps
np.mini.fe
raw TCPMPICHLAM-MPIMPI/ProMP_LitePVMTCGMSG
10.0381610.0381450.0381180.0381320.0381260.0381270.038136
20.0763020.0762610.0762670.0762680.0762830.0763010.076288
30.1144530.1144140.1144150.1143730.1144250.1144050.114432
40.1525440.1527130.152450.1526710.1525650.1526170.152642
60.2289820.2287530.2286350.2287520.2289690.2289180.228721
80.3053960.305290.3054070.3051840.3052490.3051440.305232
120.4581630.4574640.4580360.4577430.4579850.4575690.457736
130.4958890.4956890.4961750.495670.4958270.4958610.496135
160.6116420.6103650.6106370.6103760.6105160.610290.610336
190.7249960.7250920.7245420.724540.7246820.7247130.724769
210.8012740.8010740.8010890.8013380.8013070.8010360.801041
240.9154480.9155140.9155440.9154860.915530.9153080.91528
271.029591.0299481.0299541.0290441.0298171.0295611.029977
291.1080891.1063371.1067961.1064071.1060731.107121.106242
321.2216741.2207491.2201671.2208651.2205061.2200661.221164
351.3355461.3345961.3361011.3356121.3358591.3349881.335104
451.7178291.7159821.7171491.7167461.7169511.7164281.717095
481.8309071.8310431.8315881.8315111.830781.8308571.832572
511.9459031.9454451.9449291.9465781.9457791.9435781.947142
612.3280232.3270962.3270842.3269772.3265852.3303012.326965
642.4413662.4415552.4424812.4413572.4410192.4410292.442341
672.5575852.5558972.5568792.5541022.555442.5545782.555708
933.5487383.5477343.5465583.5480113.5471013.5473113.54747
963.661793.6620323.6588813.659893.6615713.6617033.661912
993.7779913.7765063.7765393.776363.7748653.7750113.777633
1254.7681634.7710534.7686354.7687394.7676354.7652484.773444
1284.8864644.8831864.8837984.8828524.8800534.8850234.884467
1314.9968794.9979834.9993034.9971944.9964454.998654.997509
1897.2092517.2128127.2151137.1985047.2077757.2090277.224662
1927.3236257.3214957.3263967.3094587.3230977.3212077.325907
1957.4424747.4253857.440837.4346687.4420077.4289437.438439
2539.6522379.6130089.6570019.653359.6512389.6225369.651957
2569.7648029.7657829.7668819.7611929.7639399.7646039.76535
2599.8794369.8764959.880429.8759839.8784159.8790439.886943
38114.53867414.53416814.53998814.337214.53245314.53045414.550331
38414.65129714.64826214.64839214.64251714.51142312.94812414.643259
38714.76172914.76257214.76274714.76221614.76055912.79933314.761997
50919.41578719.35014719.42957817.76957416.62942918.93884719.439401
51219.53775319.53164519.5237414.19078319.52795614.57332419.530778
51519.64435519.64647319.64608415.98338819.64248319.63331319.645301
76529.1912329.04698729.1945821.75095829.14253829.17941729.181284
76829.28500729.234929.30567216.38225827.52674321.30449129.319403
77129.41757629.31410229.45549723.78372924.50057329.39642129.409113
102138.99019838.95085338.97323727.9363938.93946433.9274538.855765
102439.06016239.06327839.08003835.43551839.05562234.02910739.072867
102739.17397139.1935439.13162233.33621337.39399928.96759839.174591
153340.42404541.29397553.02596952.99856542.64145955.76669444.715573
153658.59628258.59549841.68271739.30369746.2875246.17699443.108397
153942.55584744.88391334.823830.90072558.60732241.35930746.706283
204560.33208262.4131156.40987340.11293456.39788378.00530163.633343
204877.43879860.79363558.09532670.24980665.05398165.25382978.124246
205170.88820772.12204458.96681264.54488458.53885559.55146760.235903
3069105.31351978.95525277.433515106.92361589.23702875.658416108.19779
307261.55672980.9819394.68205780.902905110.51769174.05344483.082819
3075100.618714100.843534108.88967172.252514117.27657281.175949109.38065
4093150.52657190.919694116.653497100.196237103.1209881.03800394.941585
4096127.827082104.68865129.619563112.914765142.38791480.156104156.220141
4099145.13887398.204626116.486837120.49974397.48576989.868656138.246904
6141127.383841138.892796128.619426117.590748117.973491121.26169145.545558
6144181.174485127.791055131.301311138.860339176.990375135.233866185.499308
6147129.835391118.53153152.884766150.314208149.432742118.779276230.62151
8189180.634156154.084244203.064639153.702312160.12104154.436174199.179803
8192174.68441153.942764204.330629153.703529161.0131153.736802157.535419
8195227.98184156.797503157.511262154.007173155.925881154.496155159.716391
12285225.636208233.632903226.578122230.726719227.896725231.194076228.916008
12288226.724455232.644153232.228915231.852167230.743621228.647683228.720924
12291226.728076228.594239227.780522229.009944228.378106228.977752226.109776
16381274.027817269.996498278.828066179.231819284.26482151.900991204.22593
16384277.982945276.351733280.036482179.809828298.113839151.71794165.187003
16387283.153327290.893004293.326027167.589783295.438224149.266285194.67264
24573359.192948282.308403217.982385150.260275301.604907138.479871193.443439
24576340.159738281.911633212.810675185.411671302.483101146.782264198.480244
24579353.768143285.559468218.5164166.170952300.548004149.177851198.810479
32765351.665585298.956127218.861579167.213347324.164554185.567085207.638848
32768352.58368298.694194192.81117166.452061331.36703171.02341197.471557
32771352.558225298.940859209.847921166.116185328.275981173.483823223.991818
49149380.88954337.75593227.562565175.649182382.758364158.354173212.481713
49152393.521036339.017825234.7294175.347077383.206287175.782227213.439091
49155399.47588340.433648228.860379205.706406385.353738181.838338215.478994
65533431.790947354.198557258.153917203.928035404.706746172.823034211.890004
65536435.09208354.918884251.96351191.906937404.334847168.731342210.615019
65539431.57609355.824888217.435538191.047828404.319383160.425975205.46014
98301468.765745366.325772242.374732202.360242440.321872141.959444216.044434
98304465.535377366.756995232.581276206.707359439.660667156.283084217.783247
98307466.830917367.14946230.972724210.697712439.836357178.403186221.781306
131069489.291443345.743653245.804565208.269152452.386669187.23302225.34062
131072489.482042347.012124244.922947207.366292454.11793187.256014218.400171
131075489.573917346.881379238.217864204.455973450.701848159.658811213.730804
196605508.635057371.103069244.949404210.622987465.000979158.873301215.751186
196608509.192669369.845791256.20503206.692665467.058584152.366628219.242509
196611510.653796370.128594261.274386218.483467465.63005171.929814219.242179
262141519.250495374.481988250.074219227.347614480.507328193.309226216.127069
262144518.899329378.16968250.487368227.165802479.198842193.938595223.008491
262147519.862082376.559703254.481736222.787036480.306937189.632506216.962009
393213528.946367376.746706269.960148229.009271491.955472172.189377229.949015
393216528.862536377.886078260.652606228.316978492.637716152.419228229.654106
393219528.850547376.280992260.235466231.023518491.916126175.261832224.686418
524285536.336321381.906719264.72252240.330672499.874496193.202924221.069087
524288535.795764383.392169265.781876237.849558500.014481185.231732227.453615
524291535.325936381.979839263.210258241.231965500.467805174.561879236.505843
786429539.890965396.546122260.627791241.693723479.873969164.869483239.104815
786432539.000543394.814811271.640694243.593383480.917512186.811975228.067488
786435539.783632393.624866276.810349242.460141480.066764190.284062222.706801
1048573543.273793400.516715267.08372242.294513490.194555167.17891231.057719
1048576542.786753400.374608277.078206243.552917491.21092185.975476234.834827
1048579543.346043398.107209269.754803237.199879491.630663191.492171226.092428
1572861545.049287402.142993270.437248236.900217498.11037174.897829237.208959
1572864545.238477402.332566270.114694243.542479498.10248188.466939239.285902
1572867545.113937401.885761269.094139247.621159497.817105176.994108233.896438
2097149546.336638404.973404274.842188249.018302503.096422187.048445230.26663
2097152546.628151404.84732269.798585244.891435503.526966188.930202226.257586
2097155546.346304404.582477267.452855242.786406503.356768183.256671230.949342
3145725547.66063407.674074268.646931251.142982507.58838190.032713243.539449
3145728547.249052408.122113279.05632244.323469507.446331193.622106241.340654
3145731547.299578407.603619281.26667242.413983507.485904179.115823244.265444
4194301548.556058409.868781276.949382247.657302510.078797191.512844242.37043
4194304548.308041409.063322273.889194245.47475509.655568192.755163243.887175
4194307548.371515408.070974277.321616248.626628509.817165189.158894241.627226
6291453549.383346411.255734277.619735250.95497512.06178188.311532246.143897
6291456549.132201411.299219274.050815248.575501512.168575185.084858250.358308
6291459549.071005409.721312281.301741248.521657511.807629181.936362250.065428
8388605549.587164411.011102276.236099246.914032513.202478186.431618247.296077
8388608549.75903411.306707276.982361249.116277512.940591186.61904250.734921
8388611549.516123409.421407277.877982247.577508512.938407183.386672241.921636
np.mini.fe
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
000000
MPICH
MP_Lite
raw TCP
PVM
LAM/MPI
MPI/Pro
raw TCP
MPICH
LAM-MPI
MPI/Pro
MP_Lite
PVM
Message size in Bytes
Throughput in Mbps
AMD Opteron 800 HPCProcessing NodeHPC StrengthsFlat SMP like Memory Model:All four reside with the same 248 memory mapExpandable to 8P NUMA
Glue-less Coherent multi-processing:low Latency and high Bandwidth ~1600M T/sec (6.4 GB/s)
32GB of High B/W external memory bus (>5.3GB/sec.)
Native high B/W memory map I/O (>25Gbits/sec.)
Sufficiently Uniform Memory Organization (SUMO)Disadvantages3P and 4P nodes work better if the OS is aware of the memory map>4P may require a NUMA (Non-uniform memory architecture) aware OS if the CACHE hit rate is low2.6.9 kernel needed to take full advantage of NUMA architecture of OpteronsAdvantages Software view of memory is SMPLatency difference between local & remote memory is a function of the number of processors in the node1P and 2P look like a SMP machine3P and 4P are NUMA like but can still be viewed as a ccUMA or asymmetric SMP node>4P can be viewed as ccUMA and depending on CACHE hit rate, may or may not required NUMA aware OSPhysical address space is flat and can be viewed as fully coherent or not (MOEIS state)DRAM can be contiguous or interleavedAdditional processor nodes bring true increased memory bandwidthDesigned for lower overall system chip count (glue-less interface)
Future NUMA Systems Scaling beyond 8 ProcessorScaling beyond 8P is enabledExternal Coherent HyperTransport switch Coherent Interconnect Snoop filter Data cachingUp to 16 processors within the same 240 SPM memory spaceInterconnect Fabric
High Density HPC ClusterSprayCool Technology from ISR 16 cards
16G-flops/card
256G-flops peak throughput
64GB of memory per card
1TerraByte of sys. Memory
240 cubic inches114M-flops/cubic inch4.27GB of memory storage cubic inch
~6K watts ~3 watts/cubic inch141016
AMD OpteronBeowulf 4P SMP Processing NodeAMD Opteron200-333MHz9 byte Reg. DDR8GB DRAMAMD Opteron200-333MHz9 byte Reg. DDRFLASHLPCLegacy PCIUSB1.0AC97UDMA133MII10/100 Phy100 BaseTManagement LANManagementSIOPCIGraphicsVGAAMD-8111TMI/O Hub16x16 HyperTransport @ 1600MT/sPCI-XPCI-XAMD-8131TMPCI-X TunnelTo AMD 8131 TunnelTo AMD 8131 TunnelOne 4P SMP node16G-flops32GB DRAM10GB/sec. Memory BW
Single Instruction, Multiple Data (SIMD)
Extending 32-bit instruction setIntel and AMD scheme very similar48-bit virtual address space64-bit General Purpose RegistersSupport 64-bit addressing and integer mathEight extra GPR addedEight extra XMM addedDifferenceEM64T supports SSE3 instructions, Opteron has 3DNow!
Status of 64bit Linux64-bit Linux OS is a stable operating platformOpteron CPU and associated platforms have sufficient reliabilityOpteron CPU gives slightly better performance for significantly less power draw as Xeon.Using 64-bit compilation and optimization can lead to significant performance gains on AMD and Intel.
Increased Memory for 32-bit Applications32-bit server, 4 GB DRAM64-bit server, 12 GB DRAMShared256 TB32-bit App0 GB4 GB0 GB4 GB32-bit App256 TBNotsharedNotsharedNotshared OS & App share small 32-bit VM space 32-bit OS & applications all share 4GB DRAM Leads to small dataset sizes & lots of paging App has exclusive use of 32-bit VM space 64-bit OS can allocate each application large dedicated portions of 12GB DRAM OS uses VM space way above 32-bits Leads to larger dataset sizes & reduced paging
CompilersOpteron 250Legacy executable, i386: 1290 VUPSGcc 3.4.2 optimized: 2440 VUPSPathscale compiler: 2677 VUPSXeon 3.6Legacy executable, i386: 1386 VUPSGcc 3.4.2 optimized: 2309 VUPSIntel 8.1 compiler:2910 VUPSIntel 8.1 compiler with profile feedback: 4332 VUPSIntel Fortran and C 8.1 uses SSE3 instructions to optimize, makes it incompatible with Opterons.For comparison PentiumIII 1.0 GHz=568 VUPS.
64-bit Compilers
Computing Strategy: x86-64Legacy: 32-bit OSBoth AMD Athlon 64 and AMD Opteron processors run any 32-bit legacy O/SCompatible all legacy Drivers, OS & BIOSNo application recompile required, no emulation layer64-bit OSDesired applications can be written/ported to leverage the full 64-bit capabilities of x86-64Migrate only where warranted, and at the users pace32-bit applications run under 64-bit OSBIOS is standard x86 32-bit code. Transfer to 64-bit operation occurs under OS load/startup control64-bit mode does not use segmentation - Flat addressing
Compatibility Thunking LayerAMD64 Operating SystemAMD64 Device DriversAMD64 Application64-bit ProcessThunking LayerIA32 Application64-bit Process
64-bit OS & Application Interaction32-bit Compatibility Mode64-bit OS runs existing 32-bit APPs with leading edge performanceNo recompile required, 32-bit code directly executed by CPU64-bit OS provides 32-bit libraries and thunking translation layer for 32-bit system calls. 64-bit ModeMigrate only where warranted, and at the users pace to fully exploit AMD6464-bit OS requires all kernel-level programs & drivers to be ported.Any program that is linked or plugged in to a 64-bit program (ABI-level) must be ported to 64-bits.
Problems 64/32In 64bit Linux you can't run binary only programs which are compiled for IA32 or applications which haven't been ported to AMD64 yet (e.g. OpenOffice.org). This is because you can't mix 32bit applications and 64bit libraries. You would also need the 32bit versions of the libraries to run a 32bit application.Multiarcharchitectures like sparc64 or powerpc64, which provide lib for default 32bit libraries and lib64 for extra 64bit libraries, default to executing 32bit applicationsamd64 defaults to 64bit binaries because of the performance benefits it offers in 64bit mode. Thus, not wanting to rewrite virtually every binary-arch package's creation rules to install libs in lib64 not lib, and wanting to find a solution for all multiarch capable platforms, various people are working on so-called multiarch support.
Sun Solaris 64bitThe Solaris Operating System supports both 32-bit and 64-bit hardware. Customers with 32-bit hardware can run the Solaris Operating System and take advantage of the many features in the Solaris Operating System that are not explicitly related to 64-bits (e.g., dynamic reconfiguration, scalability enhancements, performance improvements). Customers can run a 32-bit application on 64- or 32-bit hardware with the Solaris Operating System without any change to the application.
Note that Solaris for x86, prior to Solaris 10, supports only a 32 bit kernel.
MacOS X xcodeAt the heart of Xcode 2.0 is Apples version of gcc 3.5, the next generation of the industry-standard gcc compiler. The new compiler helps you get more performance from your existing code by using a number of advanced optimization techniques. Auto-vectorization, a technique borrowed from the world of supercomputing, helps you to unlock the power of the Velocity Engine in every PowerPC G4 and G5 system without writing vectorized code.
With the new 64-bit support in Mac OS X Tiger, Xcode gives you the ability to create applications such as computation and rendering engines that use 64-bit memory addressing. This is ideal for data-intensive applications, which can run faster by accessing data in memory, rather than via disk access. Xcode gives you the tools for building and debugging 64-bit applications for PowerPC G5 and Mac OS X Tiger, as well as letting you create Fat Binaries that contain both 32-bit and 64-bit executables.
Kernel Overview
Kernel sourceDownload source from www.kernel.org or Mirrors.Unpack:cd /usr/srctar xzvf linux-.tar.gzln s linux-.tar.gz linuxSource-Root:/usr/src/linux
Kernel source tree. | |-- isdn | |-- asm-ppc|-- Documentation | |-- macintosh | |-- asm-ppc64|-- arch | |-- net | |-- asm-sparc| |-- alpha | |-- parport | |-- asm-sparc64| |-- arm | |-- pci | |-- asm-um| |-- i386 | |-- pcmcia | |-- asm-x86_64| | |-- boot | |-- pnp | |-- linux| | | |-- compressed | |-- scsi | |-- math-emu| | | `-- tools | |-- sgi | |-- net| | |-- kernel | |-- sound | |-- pcmcia| | |-- lib | |-- usb | |-- scsi| | |-- math-emu | `-- video | `-- video| | `-- mm |-- fs |-- init| |-- ia64 | |-- autofs |-- ipc| |-- m68k | |-- ext2 |-- kernel| |-- ppc | |-- ext3 |-- lib| |-- ppc64 | |-- fat |-- mm| |-- sparc | |-- isofs |-- net| |-- sparc64 | |-- minix | |-- 802| |-- um | |-- msdos | |-- appletalk| `-- x86_64 | |-- ntfs | |-- atm|-- drivers | |-- reiserfs | |-- bluetooth| |-- acpi | |-- smbfs | |-- core| |-- atm | |-- udf | |-- ethernet| |-- block | `-- vfat | |-- ipv4| |-- bluetooth |-- include | |-- ipv6| |-- cdrom | |-- asm -> asm-um | |-- ipx| |-- char | |-- asm-alpha | |-- irda| |-- hotplug | |-- asm-arm | |-- packet| |-- ide | |-- asm-i386 | |-- unix| |-- ieee1394 | |-- asm-ia64 | `-- x25| |-- input | |-- asm-m68k `-- scripts
Linux Source Tree Layout
/usr/src/linuxDocumentationarchfsinitkernelincludeipcdriversnetmmlibscriptsalphaarmi386ia64m68kmipsmips64ppcs390shsparcsparc64acornatmblockcdromchardiofc4i2ci2oideieee1394isdnmacintoshmiscnetadfsaffsautofsautofs4bfscodecramfsdevfsdevptsefsext2fathfshpfsasm-alphaasm-armasm-genericasm-i386asm-ia64asm-m68kasm-mipsasm-mips64linuxmath-emunetpcmciascsivideo adfsaffsautofsautofs4bfscodecramfsdevfsdevptsefsext2fathfshpfs 802appletalkatmax25bridgecoredecneteconetethernetipv4ipv6ipxirdakhttpdlapb
Sizes (linux-2.4.0-test2)sizedirectoryentriesfiles loc90M/usr/src/linux/1976452.6M4.5MDocumentation97380 na16.5March121685466K54Mdrivers3122561.5M5.6Mfs70489150K14.2Minclude192262285K28Kinit221K120Kipc664.5K332Kkernel252512K80Klib882K356Kmm191912K5.8Mnet33453162K400Kscripts264212K
linux/archSubdirectories for each current port.Each contains kernel, lib, mm, boot and other directories whose contents override code stubs in architecture independent code.lib contains highly-optimized common utility routines such as memcpy, checksums, etc.arch as of 2.4:alpha, arm, i386, ia64, m68k, mips, mips64.ppc, s390, sh, sparc, sparc64.
linux/driversLargest amount of code in the kernel tree (~1.5M).device, bus, platform and general directories.drivers/char n_tty.c is the default line discipline.drivers/block elevator.c, genhd.c, linear.c, ll_rw_blk.c, raidN.c.drivers/net specific drivers and general routines Space.c and net_init.c.drivers/scsi scsi_*.c files are generic; sd.c (disk), sr.c (CD-ROM), st.c (tape), sg.c (generic).General: cdrom, ide, isdn, parport, pcmcia, pnp, sound, telephony, video.Buses fc4, i2c, nubus, pci, sbus, tc, usb.Platforms acorn, macintosh, s390, sgi.
linux/fsContains:virtual filesystem (VFS) framework.subdirectories for actual filesystems.vfs-related files:exec.c, binfmt_*.c - files for mapping new process images.devices.c, blk_dev.c device registration, block device support.super.c, filesystems.c.inode.c, dcache.c, namei.c, buffer.c, file_table.c.open.c, read_write.c, select.c, pipe.c, fifo.c.fcntl.c, ioctl.c, locks.c, dquot.c, stat.c.
linux/includeinclude/asm-*:Architecture-dependent include subdirectories.include/linux:Header info needed both by the kernel and user apps.Usually linked to /usr/include/linux.Kernel-only portions guarded by #ifdefs#ifdef __KERNEL__ /* kernel stuff */#endifOther directories:math-emu, net, pcmcia, scsi, video.
linux/initJust two files: version.c, main.c.version.c contains the version banner that prints at boot.main.c architecture-independent boot code.start_kernel is the primary entry point.
linux/ipcSystem V IPC facilities.If disabled at compile-time, util.c exports stubs that simply return ENOSYS.One file for each facility:sem.c semaphores.shm.c shared memory.msg.c message queues.
linux/kernelThe core kernel code.sched.c the main kernel file:scheduler, wait queues, timers, alarms, task queues.Process control:fork.c, exec.c, signal.c, exit.c etcKernel module support:kmod.c, ksyms.c, module.c.Other operations:time.c, resource.c, dma.c, softirq.c, itimer.c.printk.c, info.c, panic.c, sysctl.c, sys.c.
linux/libkernel code cannot call standard C library routines.Files:brlock.c Big Reader spinlocks.cmdline.c kernel command line parsing routines.errno.c global definition of errno.inflate.c gunzip part of gzip.c used during boot.string.c portable string code.Usually replaced by optimized, architecture-dependent routines.vsprintf.c libc replacement.
linux/mmPaging and swapping:swap.c, swapfile.c (paging devices), swap_state.c (cache).vmscan.c paging policies, kswapd.page_io.c low-level page transfer.Allocation and deallocation:slab.c slab allocator.page_alloc.c page-based allocator.vmalloc.c kernel virtual-memory allocator.Memory mapping:memory.c paging, fault-handling, page table code.filemap.c file mapping.mmap.c, mremap.c, mlock.c, mprotect.c.
linux/scripts Scripts for:Menu-based kernel configuration.Kernel patching.Generating kernel documentation.
Linux Kernel ConfigurationDownload the source code and extract it under /usr/srcCustomize kernel configurationmake configmake menuconfigmake xconfigTwo menu items: Networking options and Network device supportDevice supportThree options: y, m, n
Kernel ConfigGet sources from kernel.orgUnpack in your home directorygzip -cd linux-2.4.XX.tar.gz | tar xvf -make menuconfigmake depmake bzimage
Building and Installing KernelCompiling the kernel.make dep [ make clean ]make bzImagemake modulesInstalling the kernel.make modules_install[ make install ]Kernel stored in /usr/src/linux-2.4/arch/i386/boot/bzImage and copied to /bootcp arch/i386/boot/bzImage /boot/Edit your lilo.conf and run /sbin/liloReboot.
LILO v.s. GRUBLILORun LILO to modify mini-bootloader in the MBRCannot read file system itselfGRUBMultistage loaderCan read file-system itself
Parameter passing (runlevel, init) to kernelActually hacking modifies address and name inside kernel for the process to start
Linux Loader (LILO)# sample /etc/lilo.confboot = /dev/hda delay = 40 password=SOME_PASSWORD_HERE default=vmlinuz-stable vga = normal root = /dev/hda1 image = vmlinuz-2.5.99 label = net test kernel restrictedimage = vmlinuz-stable label = stable kernel restricted other = /dev/hda3 label = Windows 2000 Professional restricted table = /dev/hda
GRUB # /etc/grub.conf generated by anacondatimeout=10splashimage=(hd0,1)/grub/splash.xpm.gzpassword --md5 $1$pd$J08sMAcfyWW.C3soZpHkh.title Red Hat Linux (2.4.18-3custom) root (hd0,1) kernel /vmlinuz-2.4.18-3custom ro root=/dev/hda5 initrd /initrd-2.4.18-3.imgtitle Red Hat Linux (2.4.18-3) Emergency kernel (no afs) root (hd0,1) kernel /vmlinuz-2.4.18-3 ro root=/dev/hda5 initrd /initrd-2.4.18-3.imgtitle Windows 2000 Professional rootnoverify (hd0,0) chainloader +1
/bootunix grub # more grub.confdefault 0# How many seconds to wait before the default listing is booted.timeout 5
title=gentoo root (hd0,0) kernel /kernel-2.4.26-gentoo-r9 root=/dev/ram0 init=/linuxrc ramdisk=8192 real_root=/dev/hda2 initrd /initrd-2.4.26-gentoo-r9
title GNU/Linux (2.4.25) root (hd0,4) kernel (hd0,4)/boot/kernel-2.4.25_pre6-gss root=/dev/ram0 init=/linuxrc real_root=/dev/hda5 vga=0x317 splash=verbose initrd (hd0,4)/boot/initrd-2.4.25_pre6-gss
unix grub # unix boot # pwd; ls -lRp/boot.:total 1582lrwxrwxrwx 1 root root 1 Sep 23 14:11 boot -> ./drwxr-xr-x 2 root root 1024 Sep 23 15:34 grub/-rw-r--r-- 1 root root 458622 Sep 23 14:58 initrd-2.4.26-gentoo-r9-rw-r--r-- 1 root root 1137878 Sep 23 14:50 kernel-2.4.26-gentoo-r9drwx------ 2 root root 12288 Sep 23 13:49 lost+found/
./grub:total 846-rw-r--r-- 1 root root 30 Sep 23 15:34 device.map-rw-r--r-- 1 root root 11264 Sep 23 15:34 e2fs_stage1_5-rw-r--r-- 1 root root 10256 Sep 23 15:34 fat_stage1_5-rw-r--r-- 1 root root 9216 Sep 23 15:34 ffs_stage1_5-rw-r--r-- 1 root root 245 Sep 23 15:34 grub.conf-rw-r--r-- 1 root root 1495 Sep 23 15:32 grub.conf.sample-rw-r--r-- 1 root root 11456 Sep 23 15:34 jfs_stage1_5lrwxrwxrwx 1 root root 9 Sep 23 15:32 menu.lst -> grub.conf-rw-r--r-- 1 root root 9600 Sep 23 15:34 minix_stage1_5-rwxr-xr-x 1 root root 196836 Sep 23 15:32 nbgrub-rwxr-xr-x 1 root root 197860 Sep 23 15:32 pxegrub-rw-r--r-- 1 root root 12864 Sep 23 15:34 reiserfs_stage1_5-rw-r--r-- 1 root root 33856 Sep 23 15:32 splash.xpm.gz-rw-r--r-- 1 root root 512 Sep 23 15:34 stage1-rw-r--r-- 1 root root 135148 Sep 23 15:34 stage2-rwxr-xr-x 1 root root 196900 Sep 23 15:32 stage2.netboot-rw-r--r-- 1 root root 8896 Sep 23 15:34 vstafs_stage1_5-rw-r--r-- 1 root root 12840 Sep 23 15:34 xfs_stage1_5
./lost+found:total 0unix boot #
Build kernel for other platformunix linux # make menuconfig ARCH=x86_64/usr/src/linux/archalpha cris ia64 mips parisc ppc64 s390x sh64 sparc64 arm i386 m68k mips64 ppc s390 sh sparc x86_64Cross-compile make HOSTCC="gcc -m32" ARCH="x86_64" bzImageget a gcc wrapper in order to crosscompile on a i386 hosthttp://www.jukie.net/~bart/debian/amd64/scripts/gcc.bart
Linux modulesdriver modules in /lib/module[root@dafinn net]# pwd; ls/lib/modules/2.4.22-1.2166.nptlsmp/kernel/drivers/net3c509.o b44.o eepro100.o netconsole.o pppox.o tg3.o3c59x.o bonding epic100.o ns83820.o ppp_synctty.o tlan.o8139cp.o de4x5.o ethertap.o pcmcia r8169.o tulip8139too.o dl2k.o fealnx.o pcnet32.o sis900.o tun.o82596.o dmfe.o irda ppp_async.o sk98lin typhoon.o8390.o dummy.o mii.o ppp_deflate.o slhc.o via-rhine.oacenic.o e100 natsemi.o ppp_generic.o smc9194.o wirelessamd8111e.o e1000 ne2k-pci.o pppoe.o starfire.oRecompiling the kernelMake the kernel smallerAdd a new deviceModify a system parameter
Hello World !#define MODULE#include int init_module() { printk("Hello, world\n"); return 0; }void cleanup_module() { printk("Goodbye !\n"); }
Loaded and Unloadedroot# gcc -c hello.croot# insmod ./hello.oHello, worldroot# rmmod helloGoodbye !root#MUST be root to load/unload moduleCheck /var/log/messages if nothing shownCan use module_init(my_init) & module_exit(my_cleanup) in Linux 2.2 and later
BSD
Source Code ControlThe entire source code for FreeBSD is stored in a CVS repositoryThe logs, and individual changes for each file can be traced back to 1994.The source tree can be checked out at any state, or corresponding to any releaseCDs are available taking the history back a further 20 years
Building WorldThe entire operating system, including all libraries and utilities can be built with a single command : make worldThe source code for the system is placed in /usr/src during installation.Much easier to secure a system if a bug is found in a key library like OpenSSL.More information in build(7) and Handbook.
Building ReleasesYou can even build a complete release of FreeBSD, including FTP install directories, floppy images, and ISO images for CDROMs with one command.make release is used by many large companies to produce special versions of FreeBSD with special patches or additional software installed by default.It is also the well documented way in which the release engineering team makes all official releases of FreeBSD.
Release Engineeringmake release makes it much easier to deploy thousands of systems pre-configured for a specific environment.The release engineering team for FreeBSD publishes schedules, identifies QA issues that must be resolved before release, and publishes documents to help other people build FreeBSD based products.See release(7) and www.freebsd.org/releng
Running a FreeBSD binaryCode likefd = open(/etc/passwd, O_RDONLY);Becomessyscall(5, ...)Kernel knows its a FreeBSD binary, uses freebsd_syscalls[ ] arrayfreebsd_syscalls[5] = freebsd_open();File is opened
Running a Linux binaryCode likefd = open(/etc/passwd, O_RDONLY);Becomessyscall(5, ...)Kernel knows its a Linux binary, uses linux_syscalls[ ] arraylinux_syscalls[5] = linux_open();File is openedAll Linux file operations redirected to /compat/linux first
Linux modulesList installed moduleslsmodModule dependenciesThe meaning of (autoclean)Load the module mannually: insmod [k] 3c509modprobe smc-ultraGenerate the dependencydepmod aRemove module: rmmodDevice driver can be dynamically loaded to compiled into the kernel.
Function pointers
LKM Utilities cmdinsmod Insert an LKM into the kernel.rmmod Remove an LKM from the kernel.depmod Determine interdependencies between LKMs.kerneld Kerneld daemon programksyms Display symbols that are exported by the kernel for use by new LKMs.lsmod List currently loaded LKMs.modinfo Display contents of .modinfo section in an LKM object file.modprobe Insert or remove an LKM or set of LKMs intelligently. For example, if you must load A before loading B, Modprobe will automatically load A when you tell it to load B.
Two ways for loading a modulemanual way : using insmod command.
automatic way : the kernel discovers the need for loading a module (for example, user mounts a file system) => requests kerneld daemon to load the needed module. => kerneld loads module using insmod.
The insmod mechanism for loading a module
reads the module into its virtual memory fixes up unresolved references to kernel routines and resources using the exported symbols from the kernel.requests the kernel for enough space to hold the new kernelthe kernel allocates a new module data structure and enough kernel memory to hold the new module and puts it at the end of the kernel modules list
The insmod mechanism for loading a module
insmod copies the module into the allocated space and relocates it so that it will run from the kernel address that it has been allocatedthe new module exports sysmbols to the kernel and insmod builds a table of these exported symbolsif the new module depends on another module, that module has the reference of the new module.the kernel calls the modules initialization routine and carries on installing the module
Unloadig a moduleTwo ways for unloading a moduleManual way : uses rmmod commandautomatic way : when idle timer expires, the kerneld calls the service routines for all unused loaded modulesthe mechanism of unloadingif the module can be unloaded, its cleanup routine is called to free up the kernel resources that it has allocatedthe module data structure is unlinked from the list of kernel modulesall of the kernel memory that the module needed is deallocated.
User Space DriverAdvantages Full C library can be linked inRun in conventional debuggerProblem in driver unlikely to hang entire systemUser memory is swapping (more to use)Still allow concurrent access to a device for well-designed driverDisadvantages Interrupts are not available in user spaceDirect access to memory by mmapping /dev/mem (only privileged user)Access to I/O ports after calling ioperm or iopl (not all platform support), and access to /dev/port can be too slow (only privileged user)Response time is slower (context switch, driver swapped to disk)Most important devices cant be handled in user space (block, network)
The Pentium III 1.0 GHz came out 4 years ago (June 2001).Thunking is a term used to describe calling cross platform code and can mean calling 32 or 16bit code from a 64bit platform or calling 64bit code from a 32bit platform. In essence, a thunk is a translation layer that sits between the 32bit application and the 64bit process code.There are two basic methods used for thunking, which is a function of the calling platform. In the case of 32bit application calling 64bit application, the generic thunking functions in the 64bit kernel can be used. For the opposite scenario, the 64bit process must use a flat thunk to get the job done. It is also worth noting that flat thunks can be used to make 32bit to 64bit calls but because flat thunks are far more difficult, generic thunks are preferred for this situation.Process management creating and destroying processes and handling heir connection to the outside world. Process communication and scheduler.
Memory management builds up a virtual addressing space. Different parts of kernel interact with malloc/free
Filesystems almost everything can be treated as a file. Build a structured filesystem on top of unstructured hardware. Support multiple filesystems such as ext2, FAT.
Device Control kernel must have embedded in its a device driver for every peripheral present on a system.
Networking most network operations are not specific to a process, incoming packets are asynchronous events.
MODULE is needed except for drivers that are linked directly to the kernel.
Printk() is defined in Linux kernel and behaves similarly to printf. The kernel runs by itself without the help of C library. No floating point support before Linux 2.2The string is the priority of the message so it can show on console. (depending on configuration of klogd daemon)
static int __init my_init(void)static void __exit my_cleanup(void)Advantages: (i)Each init and cleanup have unique name to helps debugging.(ii) Make life easier for those writing drivers that work either as a module or built directly into the kernel (no need to change function name)
User program that RW directly to device portsBlock device, network interfacesExample is X server, gpm mouse driver