forked from ekoontz/hadoop-common
-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathCHANGES.txt
10906 lines (7441 loc) · 423 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Hadoop Change Log
Trunk (unreleased changes)
INCOMPATIBLE CHANGES
HADOOP-6904. Support method based RPC compatiblity. (hairong)
HADOOP-6432. Add Statistics support in FileContext. (jitendra)
HADOOP-7136. Remove failmon contrib component. (nigel)
NEW FEATURES
HADOOP-7322. Adding a util method in FileUtil for directory listing,
avoid NPEs on File.listFiles() (Bharath Mundlapudi via mattf)
HADOOP-7023. Add listCorruptFileBlocks to Filesysem. (Patrick Kling
via hairong)
HADOOP-7096. Allow setting of end-of-record delimiter for TextInputFormat
(Ahmed Radwan via todd)
HADOOP-6994. Api to get delegation token in AbstractFileSystem. (jitendra)
HADOOP-7171. Support UGI in FileContext API. (jitendra)
HADOOP-7257 Client side mount tables (sanjay)
HADOOP-6919. New metrics2 framework. (Luke Lu via acmurthy)
HADOOP-6920. Metrics instrumentation to move new metrics2 framework.
(Luke Lu via suresh)
HADOOP-7214. Add Common functionality necessary to provide an equivalent
of /usr/bin/groups for Hadoop. (Aaron T. Myers via todd)
HADOOP-6832. Add an authentication plugin using a configurable static user
for the web UI. (Owen O'Malley and Todd Lipcon via cdouglas)
IMPROVEMENTS
HADOOP-7042. Updates to test-patch.sh to include failed test names and
improve other messaging. (nigel)
HADOOP-7001. Configuration changes can occur via the Reconfigurable
interface. (Patrick Kling via dhruba)
HADOOP-6764. Add number of reader threads and queue length as
configuration parameters in RPC.getServer. (Dmytro Molkov via hairong)
HADOOP-7049. TestReconfiguration should be junit v4.
(Patrick Kling via eli)
HADOOP-7054 Change NN LoadGenerator to use FileContext APIs
(Sanjay Radia)
HADOOP-7060. A more elegant FileSystem#listCorruptFileBlocks API.
(Patrick Kling via hairong)
HADOOP-7058. Expose number of bytes in FSOutputSummer buffer to
implementatins. (Todd Lipcon via hairong)
HADOOP-7061. unprecise javadoc for CompressionCodec. (Jingguo Yao via eli)
HADOOP-7059. Remove "unused" warning in native code. (Noah Watkins via eli)
HADOOP-6864. Provide a JNI-based implementation of
ShellBasedUnixGroupsNetgroupMapping
(implementation of GroupMappingServiceProvider) (Erik Seffl via boryas)
HADOOP-7078. Improve javadocs for RawComparator interface.
(Harsh J Chouraria via todd)
HADOOP-6995. Allow wildcards to be used in ProxyUsers configurations.
(todd)
HADOOP-6376. Add a comment header to conf/slaves that specifies the file
format. (Kay Kay via todd)
HADOOP-7151. Document need for stable hashCode() in WritableComparable.
(Dmitriy V. Ryaboy via todd)
HADOOP-7112. Issue a warning when GenericOptionsParser libjars are not on
local filesystem. (tomwhite)
HADOOP-7114. FsShell should dump all exceptions at DEBUG level.
(todd via tomwhite)
HADOOP-7159. RPC server should log the client hostname when read exception
happened. (Scott Chen via todd)
HADOOP-7167. Allow using a file to exclude certain tests from build. (todd)
HADOOP-7133. Batch the calls in DataStorage to FileUtil.createHardLink().
(Matt Foley via jghoman)
HADOOP-7166. Add DaemonFactory to common. (Erik Steffl & jitendra)
HADOOP-7175. Add isEnabled() to Trash. (Daryn Sharp via szetszwo)
HADOOP-7180. Better support on CommandFormat on the API and exceptions.
(Daryn Sharp via szetszwo)
HADOOP-7202. Improve shell Command base class. (Daryn Sharp via szetszwo)
HADOOP-7224. Add CommandFactory to shell. (Daryn Sharp via szetszwo)
HADOOP-7014. Generalize CLITest structure and interfaces to facilitate
upstream adoption (e.g. for web testing). (cos)
HADOOP-7230. Move "fs -help" shell command tests from HDFS to COMMOM; see
also HDFS-1844. (Daryn Sharp via szetszwo)
HADOOP-7233. Refactor ls to conform to new FsCommand class. (Daryn Sharp
via szetszwo)
HADOOP-7235. Refactor the tail command to conform to new FsCommand class.
(Daryn Sharp via szetszwo)
HADOOP-7179. Federation: Improve HDFS startup scripts. (Erik Steffl
and Tanping Wang via suresh)
HADOOP-7227. Remove protocol version check at proxy creation in Hadoop
RPC. (jitendra)
HADOOP-7236. Refactor the mkdir command to conform to new FsCommand class.
(Daryn Sharp via szetszwo)
HADOOP-7250. Refactor the setrep command to conform to new FsCommand class.
(Daryn Sharp via szetszwo)
HADOOP-7249. Refactor the chmod/chown/chgrp command to conform to new
FsCommand class. (Daryn Sharp via szetszwo)
HADOOP-7251. Refactor the getmerge command to conform to new FsCommand
class. (Daryn Sharp via szetszwo)
HADOOP-7265. Keep track of relative paths in PathData. (Daryn Sharp
via szetszwo)
HADOOP-7238. Refactor the cat and text commands to conform to new FsCommand
class. (Daryn Sharp via szetszwo)
HADOOP-7271. Standardize shell command error messages. (Daryn Sharp
via szetszwo)
HADOOP-7272. Remove unnecessary security related info logs. (suresh)
HADOOP-7275. Refactor the stat command to conform to new FsCommand
class. (Daryn Sharp via szetszwo)
HADOOP-7237. Refactor the touchz command to conform to new FsCommand
class. (Daryn Sharp via szetszwo)
HADOOP-7267. Refactor the rm/rmr/expunge commands to conform to new
FsCommand class. (Daryn Sharp via szetszwo)
HADOOP-7285. Refactor the test command to conform to new FsCommand
class. (Daryn Sharp via todd)
HADOOP-7289. In ivy.xml, test conf should not extend common conf.
(Eric Yang via szetszwo)
HADOOP-7291. Update Hudson job not to run test-contrib. (Nigel Daley via eli)
HADOOP-7286. Refactor the du/dus/df commands to conform to new FsCommand
class. (Daryn Sharp via todd)
HADOOP-7301. FSDataInputStream should expose a getWrappedStream method.
(Jonathan Hsieh via eli)
HADOOP-7306. Start metrics system even if config files are missing
(Luke Lu via todd)
HADOOP-7302. webinterface.private.actions should be renamed and moved to
the MapReduce project. (Ari Rabkin via todd)
HADOOP-7329. Improve help message for "df" to include "-h" flag.
(Xie Xianshan via todd)
HADOOP-7320. Refactor the copy and move commands to conform to new
FsCommand class. (Daryn Sharp via todd)
HADOOP-7312. Update value of hadoop.common.configuration.version.
(Harsh J Chouraria via todd)
HADOOP-7337. Change PureJavaCrc32 annotations to public stable. (szetszwo)
OPTIMIZATIONS
HADOOP-7333. Performance improvement in PureJavaCrc32. (Eric Caspole
via todd)
BUG FIXES
HADOOP-7015. RawLocalFileSystem#listStatus does not deal with a directory
whose entries are changing (e.g. in a multi-thread or multi-process
environment). (Sanjay Radia via eli)
HADOOP-7045. TestDU fails on systems with local file systems with
extended attributes. (eli)
HADOOP-6939. Inconsistent lock ordering in
AbstractDelegationTokenSecretManager. (Todd Lipcon via tomwhite)
HADOOP-7129. Fix typo in method name getProtocolSigature (todd)
HADOOP-7048. Wrong description of Block-Compressed SequenceFile Format in
SequenceFile's javadoc. (Jingguo Yao via tomwhite)
HADOOP-7153. MapWritable violates contract of Map interface for equals()
and hashCode(). (Nicholas Telford via todd)
HADOOP-6754. DefaultCodec.createOutputStream() leaks memory.
(Aaron Kimball via tomwhite)
HADOOP-7098. Tasktracker property not set in conf/hadoop-env.sh.
(Bernd Fondermann via tomwhite)
HADOOP-7131. Exceptions thrown by Text methods should include the causing
exception. (Uma Maheswara Rao G via todd)
HADOOP-6912. Guard against NPE when calling UGI.isLoginKeytabBased().
(Kan Zhang via jitendra)
HADOOP-7204. remove local unused fs variable from CmdHandler
and FsShellPermissions.changePermissions (boryas)
HADOOP-7210. Chown command is not working from FSShell
(Uma Maheswara Rao G via todd)
HADOOP-7215. RPC clients must use network interface corresponding to
the host in the client's kerberos principal key. (suresh)
HADOOP-7019. Refactor build targets to enable faster cross project dev
cycles. (Luke Lu via cos)
HADOOP-7216. Add FsCommand.runAll() with deprecated annotation for the
transition of Command base class improvement. (Daryn Sharp via szetszwo)
HADOOP-7207. fs member of FSShell is not really needed (boryas)
HADOOP-7223. FileContext createFlag combinations are not clearly defined.
(suresh)
HADOOP-7231. Fix synopsis for -count. (Daryn Sharp via eli).
HADOOP-7261. Disable IPV6 for junit tests. (suresh)
HADOOP-7268. FileContext.getLocalFSFileContext() behavior needs to be fixed
w.r.t tokens. (jitendra)
HADOOP-7290. Unit test failure in TestUserGroupInformation.testGetServerSideGroups. (Trevor Robison via eli)
HADOOP-7292. Fix racy test case TestSinkQueue. (Luke Lu via todd)
HADOOP-7282. ipc.Server.getRemoteIp() may return null. (John George
via szetszwo)
Release 0.22.0 - Unreleased
INCOMPATIBLE CHANGES
HADOOP-7137. Remove hod contrib. (nigel via eli)
NEW FEATURES
HADOOP-6791. Refresh for proxy superuser config
(common part for HDFS-1096) (boryas)
HADOOP-6581. Add authenticated TokenIdentifiers to UGI so that
they can be used for authorization (Kan Zhang and Jitendra Pandey
via jghoman)
HADOOP-6584. Provide Kerberized SSL encryption for webservices.
(jghoman and Kan Zhang via jghoman)
HADOOP-6853. Common component of HDFS-1045. (jghoman)
HADOOP-6859 - Introduce additional statistics to FileSystem to track
file system operations (suresh)
HADOOP-6870. Add a new API getFiles to FileSystem and FileContext that
lists all files under the input path or the subtree rooted at the
input path if recursive is true. Block locations are returned together
with each file's status. (hairong)
HADOOP-6888. Add a new FileSystem API closeAllForUGI(..) for closing all
file systems associated with a particular UGI. (Devaraj Das and Kan Zhang
via szetszwo)
HADOOP-6892. Common component of HDFS-1150 (Verify datanodes' identities
to clients in secure clusters) (jghoman)
HADOOP-6889. Make RPC to have an option to timeout. (hairong)
HADOOP-6996. Allow CodecFactory to return a codec object given a codec'
class name. (hairong)
HADOOP-7013. Add boolean field isCorrupt to BlockLocation.
(Patrick Kling via hairong)
HADOOP-6978. Adds support for NativeIO using JNI.
(Todd Lipcon, Devaraj Das & Owen O'Malley via ddas)
HADOOP-7134. configure files that are generated as part of the released
tarball need to have executable bit set. (Roman Shaposhnik via cos)
IMPROVEMENTS
HADOOP-6644. util.Shell getGROUPS_FOR_USER_COMMAND method name
- should use common naming convention (boryas)
HADOOP-6778. add isRunning() method to
AbstractDelegationTokenSecretManager (for HDFS-1044) (boryas)
HADOOP-6633. normalize property names for JT/NN kerberos principal
names in configuration (boryas)
HADOOP-6627. "Bad Connection to FS" message in FSShell should print
message from the exception (boryas)
HADOOP-6600. mechanism for authorization check for inter-server
protocols. (boryas)
HADOOP-6623. Add StringUtils.split for non-escaped single-character
separator. (Todd Lipcon via tomwhite)
HADOOP-6761. The Trash Emptier has the ability to run more frequently.
(Dmytro Molkov via dhruba)
HADOOP-6714. Resolve compressed files using CodecFactory in FsShell::text.
(Patrick Angeles via cdouglas)
HADOOP-6661. User document for UserGroupInformation.doAs.
(Jitendra Pandey via jghoman)
HADOOP-6674. Makes use of the SASL authentication options in the
SASL RPC. (Jitendra Pandey via ddas)
HADOOP-6526. Need mapping from long principal names to local OS
user names. (boryas)
HADOOP-6814. Adds an API in UserGroupInformation to get the real
authentication method of a passed UGI. (Jitendra Pandey via ddas)
HADOOP-6756. Documentation for common configuration keys.
(Erik Steffl via shv)
HADOOP-6835. Add support for concatenated gzip input. (Greg Roelofs via
cdouglas)
HADOOP-6845. Renames the TokenStorage class to Credentials.
(Jitendra Pandey via ddas)
HADOOP-6826. FileStatus needs unit tests. (Rodrigo Schmidt via Eli
Collins)
HADOOP-6905. add buildDTServiceName method to SecurityUtil
(as part of MAPREDUCE-1718) (boryas)
HADOOP-6632. Adds support for using different keytabs for different
servers in a Hadoop cluster. In the earier implementation, all servers
of a certain type (like TaskTracker), would have the same keytab and the
same principal. Now the principal name is a pattern that has _HOST in it.
(Kan Zhang & Jitendra Pandey via ddas)
HADOOP-6861. Adds new non-static methods in Credentials to read and
write token storage file. (Jitendra Pandey & Owen O'Malley via ddas)
HADOOP-6877. Common part of HDFS-1178 (NameNode servlets should communicate
with NameNode directrly). (Kan Zhang via jghoman)
HADOOP-6475. Adding some javadoc to Server.RpcMetrics, UGI.
(Jitendra Pandey and borya via jghoman)
HADOOP-6656. Adds a thread in the UserGroupInformation to renew TGTs
periodically. (Owen O'Malley and ddas via ddas)
HADOOP-6890. Improve listFiles API introduced by HADOOP-6870. (hairong)
HADOOP-6862. Adds api to add/remove user and group to AccessControlList
(amareshwari)
HADOOP-6911. doc update for DelegationTokenFetcher (boryas)
HADOOP-6900. Make the iterator returned by FileSystem#listLocatedStatus to
throw IOException rather than RuntimeException when there is an IO error
fetching the next file. (hairong)
HADOOP-6905. Better logging messages when a delegation token is invalid.
(Kan Zhang via jghoman)
HADOOP-6693. Add metrics to track kerberol login activity. (suresh)
HADOOP-6803. Add native gzip read/write coverage to TestCodec.
(Eli Collins via tomwhite)
HADOOP-6950. Suggest that HADOOP_CLASSPATH should be preserved in
hadoop-env.sh.template. (Philip Zeyliger via Eli Collins)
HADOOP-6922. Make AccessControlList a writable and update documentation
for Job ACLs. (Ravi Gummadi via vinodkv)
HADOOP-6965. Introduces checks for whether the original tgt is valid
in the reloginFromKeytab method.
HADOOP-6856. Simplify constructors for SequenceFile, and MapFile. (omalley)
HADOOP-6987. Use JUnit Rule to optionally fail test cases that run more
than 10 seconds (jghoman)
HADOOP-7005. Update test-patch.sh to remove callback to Hudson. (nigel)
HADOOP-6985. Suggest that HADOOP_OPTS be preserved in
hadoop-env.sh.template. (Ramkumar Vadali via cutting)
HADOOP-7007. Update the hudson-test-patch ant target to work with the
latest test-patch.sh script (gkesavan)
HADOOP-7010. Typo in FileSystem.java. (Jingguo Yao via eli)
HADOOP-7009. MD5Hash provides a public factory method that creates an
instance of thread local MessageDigest. (hairong)
HADOOP-7008. Enable test-patch.sh to have a configured number of
acceptable findbugs and javadoc warnings. (nigel and gkesavan)
HADOOP-6818. Provides a JNI implementation of group resolution. (ddas)
HADOOP-6943. The GroupMappingServiceProvider interface should be public.
(Aaron T. Myers via tomwhite)
HADOOP-4675. Current Ganglia metrics implementation is incompatible with
Ganglia 3.1. (Brian Bockelman via tomwhite)
HADOOP-6977. Herriot daemon clients should vend statistics (cos)
HADOOP-7024. Create a test method for adding file systems during tests.
(Kan Zhang via jghoman)
HADOOP-6903. Make AbstractFSileSystem methods and some FileContext methods
to be public. (Sanjay Radia)
HADOOP-7034. Add TestPath tests to cover dot, dot dot, and slash
normalization. (eli)
HADOOP-7032. Assert type constraints in the FileStatus constructor. (eli)
HADOOP-6562. FileContextSymlinkBaseTest should use FileContextTestHelper.
(eli)
HADOOP-7028. ant eclipse does not include requisite ant.jar in the
classpath. (Patrick Angeles via eli)
HADOOP-6298. Add copyBytes to Text and BytesWritable. (omalley)
HADOOP-6578. Configuration should trim whitespace around a lot of value
types. (Michele Catasta via eli)
HADOOP-6811. Remove EC2 bash scripts. They are replaced by Apache Whirr
(incubating, http://incubator.apache.org/whirr). (tomwhite)
HADOOP-7102. Remove "fs.ramfs.impl" field from core-deafult.xml (shv)
HADOOP-7104. Remove unnecessary DNS reverse lookups from RPC layer
(Kan Zhang via todd)
HADOOP-6056. Use java.net.preferIPv4Stack to force IPv4.
(Michele Catasta via shv)
HADOOP-7110. Implement chmod with JNI. (todd)
HADOOP-6812. Change documentation for correct placement of configuration
variables: mapreduce.reduce.input.buffer.percent,
mapreduce.task.io.sort.factor, mapreduce.task.io.sort.mb
(Chris Douglas via shv)
HADOOP-6436. Remove auto-generated native build files. (rvs via eli)
HADOOP-6970. SecurityAuth.audit should be generated under /build. (boryas)
HADOOP-7154. Should set MALLOC_ARENA_MAX in hadoop-env.sh (todd)
HADOOP-7187. Fix socket leak in GangliaContext. (Uma Maheswara Rao G
via szetszwo)
HADOOP-7241. fix typo of command 'hadoop fs -help tail'.
(Wei Yongjun via eli)
HADOOP-7244. Documentation change for updated configuration keys.
(tomwhite via eli)
HADOOP-7189. Add ability to enable 'debug' property in JAAS configuration.
(Ted Yu via todd)
HADOOP-7192. Update fs -stat docs to reflect the format features. (Harsh
J Chouraria via todd)
OPTIMIZATIONS
HADOOP-6884. Add LOG.isDebugEnabled() guard for each LOG.debug(..).
(Erik Steffl via szetszwo)
HADOOP-6683. ZlibCompressor does not fully utilize the buffer.
(Kang Xiao via eli)
HADOOP-6949. Reduce RPC packet size of primitive arrays using
ArrayPrimitiveWritable instead of ObjectWritable. (Matt Foley via suresh)
BUG FIXES
HADOOP-6638. try to relogin in a case of failed RPC connection (expired
tgt) only in case the subject is loginUser or proxyUgi.realUser. (boryas)
HADOOP-6781. security audit log shouldn't have exception in it. (boryas)
HADOOP-6612. Protocols RefreshUserToGroupMappingsProtocol and
RefreshAuthorizationPolicyProtocol will fail with security enabled (boryas)
HADOOP-6764. Remove verbose logging from the Groups class. (Boris Shkolnik)
HADOOP-6730. Bug in FileContext#copy and provide base class for
FileContext tests. (Ravi Phulari via jghoman)
HADOOP-6669. Respect compression configuration when creating DefaultCodec
instances. (Koji Noguchi via cdouglas)
HADOOP-6747. TestNetUtils fails on Mac OS X. (Todd Lipcon via jghoman)
HADOOP-6787. Factor out glob pattern code from FileContext and
Filesystem. Also fix bugs identified in HADOOP-6618 and make the
glob pattern code less restrictive and more POSIX standard
compliant. (Luke Lu via eli)
HADOOP-6649. login object in UGI should be inside the subject (jnp via
boryas)
HADOOP-6687. user object in the subject in UGI should be reused in case
of a relogin. (jnp via boryas)
HADOOP-6603. Provide workaround for issue with Kerberos not resolving
cross-realm principal (Kan Zhang and Jitendra Pandey via jghoman)
HADOOP-6620. NPE if renewer is passed as null in getDelegationToken.
(Jitendra Pandey via jghoman)
HADOOP-6613. Moves the RPC version check ahead of the AuthMethod check.
(Kan Zhang via ddas)
HADOOP-6682. NetUtils:normalizeHostName does not process hostnames starting
with [a-f] correctly. (jghoman)
HADOOP-6652. Removes the unnecessary cache from
ShellBasedUnixGroupsMapping. (ddas)
HADOOP-6815. refreshSuperUserGroupsConfiguration should use server side
configuration for the refresh (boryas)
HADOOP-6648. Adds a check for null tokens in Credentials.addToken api.
(ddas)
HADOOP-6647. balancer fails with "is not authorized for protocol
interface NamenodeProtocol" in secure environment (boryas)
HADOOP-6834. TFile.append compares initial key against null lastKey
(hong tang via mahadev)
HADOOP-6670. Use the UserGroupInformation's Subject as the criteria for
equals and hashCode. (Owen O'Malley and Kan Zhang via ddas)
HADOOP-6536. Fixes FileUtil.fullyDelete() not to delete the contents of
the sym-linked directory. (Ravi Gummadi via amareshwari)
HADOOP-6873. using delegation token over hftp for long
running clients (boryas)
HADOOP-6706. Improves the sasl failure handling due to expired tickets,
and other server detected failures. (Jitendra Pandey and ddas via ddas)
HADOOP-6715. Fixes AccessControlList.toString() to return a descriptive
String representation of the ACL. (Ravi Gummadi via amareshwari)
HADOOP-6885. Fix java doc warnings in Groups and
RefreshUserMappingsProtocol. (Eli Collins via jghoman)
HADOOP-6482. GenericOptionsParser constructor that takes Options and
String[] ignores options. (Eli Collins via jghoman)
HADOOP-6906. FileContext copy() utility doesn't work with recursive
copying of directories. (vinod k v via mahadev)
HADOOP-6453. Hadoop wrapper script shouldn't ignore an existing
JAVA_LIBRARY_PATH. (Chad Metcalf via jghoman)
HADOOP-6932. Namenode start (init) fails because of invalid kerberos
key, even when security set to "simple" (boryas)
HADOOP-6913. Circular initialization between UserGroupInformation and
KerberosName (Kan Zhang via boryas)
HADOOP-6907. Rpc client doesn't use the per-connection conf to figure
out server's Kerberos principal (Kan Zhang via hairong)
HADOOP-6938. ConnectionId.getRemotePrincipal() should check if security
is enabled. (Kan Zhang via hairong)
HADOOP-6930. AvroRpcEngine doesn't work with generated Avro code.
(sharad)
HADOOP-6940. RawLocalFileSystem's markSupported method misnamed
markSupport. (Tom White via eli).
HADOOP-6951. Distinct minicluster services (e.g. NN and JT) overwrite each
other's service policies. (Aaron T. Myers via tomwhite)
HADOOP-6879. Provide SSH based (Jsch) remote execution API for system
tests (cos)
HADOOP-6989. Correct the parameter for SetFile to set the value type
for SetFile to be NullWritable instead of the key. (cdouglas via omalley)
HADOOP-6984. Combine the compress kind and the codec in the same option
for SequenceFiles. (cdouglas via omalley)
HADOOP-6933. TestListFiles is flaky. (Todd Lipcon via tomwhite)
HADOOP-6947. Kerberos relogin should set refreshKrb5Config to true.
(Todd Lipcon via tomwhite)
HADOOP-7006. Fix 'fs -getmerge' command to not be a no-op.
(Chris Nauroth via cutting)
HADOOP-6663. BlockDecompressorStream get EOF exception when decompressing
the file compressed from empty file. (Kang Xiao via tomwhite)
HADOOP-6991. Fix SequenceFile::Reader to honor file lengths and call
openFile (cdouglas via omalley)
HADOOP-7011. Fix KerberosName.main() to not throw an NPE.
(Aaron T. Myers via tomwhite)
HADOOP-6975. Integer overflow in S3InputStream for blocks > 2GB.
(Patrick Kling via tomwhite)
HADOOP-6758. MapFile.fix does not allow index interval definition.
(Gianmarco De Francisci Morales via tomwhite)
HADOOP-6926. SocketInputStream incorrectly implements read().
(Todd Lipcon via tomwhite)
HADOOP-6899 RawLocalFileSystem#setWorkingDir() does not work for relative names
(Sanjay Radia)
HADOOP-6496. HttpServer sends wrong content-type for CSS files
(and others). (Todd Lipcon via tomwhite)
HADOOP-7057. IOUtils.readFully and IOUtils.skipFully have typo in
exception creation's message. (cos)
HADOOP-7038. saveVersion script includes an additional \r while running
whoami under windows. (Wang Xu via cos)
HADOOP-7082. Configuration.writeXML should not hold lock while outputting
(todd)
HADOOP-7070. JAAS configuration should delegate unknown application names
to pre-existing configuration. (todd)
HADOOP-7087. SequenceFile.createWriter ignores FileSystem parameter (todd)
HADOOP-7091. reloginFromKeytab() should happen even if TGT can't be found.
(Kan Zhang via jghoman)
HADOOP-7100. Fix build to not refer to contrib/ec2 removed by HADOOP-6811
(todd)
HADOOP-7097. JAVA_LIBRARY_PATH missing base directory. (Noah Watkins via
todd)
HADOOP-7093. Servlets should default to text/plain (todd)
HADOOP-7101. UserGroupInformation.getCurrentUser() fails when called from
non-Hadoop JAAS context. (todd)
HADOOP-7089. Fix link resolution logic in hadoop-config.sh. (eli)
HADOOP-7046. Fix Findbugs warning in Configuration. (Po Cheung via shv)
HADOOP-7118. Fix NPE in Configuration.writeXml (todd)
HADOOP-7122. Fix thread leak when shell commands time out. (todd)
HADOOP-7126. Fix file permission setting for RawLocalFileSystem on Windows.
(Po Cheung via shv)
HADOOP-6642. Fix javac, javadoc, findbugs warnings related to security work.
(Chris Douglas, Po Cheung via shv)
HADOOP-7140. IPC Reader threads do not stop when server stops (todd)
HADOOP-7094. hadoop.css got lost during project split (cos)
HADOOP-7145. Configuration.getLocalPath should trim whitespace from
the provided directories. (todd)
HADOOP-7156. Workaround for unsafe implementations of getpwuid_r (todd)
HADOOP-6898. FileSystem.copyToLocal creates files with 777 permissions.
(Aaron T. Myers via tomwhite)
HADOOP-7229. Do not default to an absolute path for kinit in Kerberos
auto-renewal thread. (Aaron T. Myers via todd)
HADOOP-7172. SecureIO should not check owner on non-secure
clusters that have no native support. (todd via eli)
HADOOP-7184. Remove deprecated config local.cache.size from
core-default.xml (todd)
HADOOP-7245. FsConfig should use constants in CommonConfigurationKeys.
(tomwhite via eli)
HADOOP-7068. Ivy resolve force mode should be turned off by default.
(Luke Lu via tomwhite)
HADOOP-7296. The FsPermission(FsPermission) constructor does not use the
sticky bit. (Siddharth Seth via tomwhite)
HADOOP-7300. Configuration methods that return collections are inconsistent
about mutability. (todd)
HADOOP-7305. Eclipse project classpath should include tools.jar from JDK.
(Niels Basjes via todd)
HADOOP-7318. MD5Hash factory should reset the digester it returns.
(todd via eli)
HADOOP-7287. Configuration deprecation mechanism doesn't work properly for
GenericOptionsParser and Tools. (Aaron T. Myers via todd)
HADOOP-7146. RPC server leaks file descriptors (todd)
HADOOP-7276. Hadoop native builds fail on ARM due to -m32 (Trevor Robinson
via eli)
Release 0.21.1 - Unreleased
IMPROVEMENTS
HADOOP-6934. Test for ByteWritable comparator.
(Johannes Zillmann via Eli Collins)
HADOOP-6786. test-patch needs to verify Herriot integrity (cos)
HADOOP-7177. CodecPool should report which compressor it is using.
(Allen Wittenauer via eli)
BUG FIXES
HADOOP-6925. BZip2Codec incorrectly implements read().
(Todd Lipcon via Eli Collins)
HADOOP-6833. IPC leaks call parameters when exceptions thrown.
(Todd Lipcon via Eli Collins)
HADOOP-6971. Clover build doesn't generate per-test coverage (cos)
HADOOP-6993. Broken link on cluster setup page of docs. (eli)
HADOOP-6944. [Herriot] Implement a functionality for getting proxy users
definitions like groups and hosts. (Vinay Thota via cos)
HADOOP-6954. Sources JARs are not correctly published to the Maven
repository. (tomwhite)
HADOOP-7052. misspelling of threshold in conf/log4j.properties.
(Jingguo Yao via eli)
HADOOP-7053. wrong FSNamesystem Audit logging setting in
conf/log4j.properties. (Jingguo Yao via eli)
HADOOP-7120. Fix a syntax error in test-patch.sh. (szetszwo)
HADOOP-7162. Rmove a duplicated call FileSystem.listStatus(..) in FsShell.
(Alexey Diomin via szetszwo)
HADOOP-7117. Remove fs.checkpoint.* from core-default.xml and replace
fs.checkpoint.* with dfs.namenode.checkpoint.* in documentations.
(Harsh J Chouraria via szetszwo)
HADOOP-7193. Correct the "fs -touchz" command help message.
(Uma Maheswara Rao G via szetszwo)
HADOOP-7174. Null is displayed in the "fs -copyToLocal" command.
(Uma Maheswara Rao G via szetszwo)
HADOOP-7194. Fix resource leak in IOUtils.copyBytes(..).
(Devaraj K via szetszwo)
HADOOP-7183. WritableComparator.get should not cache comparator objects.
(tomwhite via eli)
Release 0.21.0 - 2010-08-13
INCOMPATIBLE CHANGES
HADOOP-4895. Remove deprecated methods DFSClient.getHints(..) and
DFSClient.isDirectory(..). (szetszwo)
HADOOP-4941. Remove deprecated FileSystem methods: getBlockSize(Path f),
getLength(Path f) and getReplication(Path src). (szetszwo)
HADOOP-4648. Remove obsolete, deprecated InMemoryFileSystem and
ChecksumDistributedFileSystem. (cdouglas via szetszwo)
HADOOP-4940. Remove a deprecated method FileSystem.delete(Path f). (Enis
Soztutar via szetszwo)
HADOOP-4010. Change semantics for LineRecordReader to read an additional
line per split- rather than moving back one character in the stream- to
work with splittable compression codecs. (Abdul Qadeer via cdouglas)
HADOOP-5094. Show hostname and separate live/dead datanodes in DFSAdmin
report. (Jakob Homan via szetszwo)
HADOOP-4942. Remove deprecated FileSystem methods getName() and
getNamed(String name, Configuration conf). (Jakob Homan via szetszwo)
HADOOP-5486. Removes the CLASSPATH string from the command line and instead
exports it in the environment. (Amareshwari Sriramadasu via ddas)
HADOOP-2827. Remove deprecated NetUtils::getServerAddress. (cdouglas)
HADOOP-5681. Change examples RandomWriter and RandomTextWriter to
use new mapreduce API. (Amareshwari Sriramadasu via sharad)
HADOOP-5680. Change org.apache.hadoop.examples.SleepJob to use new
mapreduce api. (Amareshwari Sriramadasu via sharad)
HADOOP-5699. Change org.apache.hadoop.examples.PiEstimator to use
new mapreduce api. (Amareshwari Sriramadasu via sharad)
HADOOP-5720. Introduces new task types - JOB_SETUP, JOB_CLEANUP
and TASK_CLEANUP. Removes the isMap methods from TaskID/TaskAttemptID
classes. (ddas)
HADOOP-5668. Change TotalOrderPartitioner to use new API. (Amareshwari
Sriramadasu via cdouglas)
HADOOP-5738. Split "waiting_tasks" JobTracker metric into waiting maps and
waiting reduces. (Sreekanth Ramakrishnan via cdouglas)
HADOOP-5679. Resolve findbugs warnings in core/streaming/pipes/examples.
(Jothi Padmanabhan via sharad)
HADOOP-4359. Support for data access authorization checking on Datanodes.
(Kan Zhang via rangadi)
HADOOP-5690. Change org.apache.hadoop.examples.DBCountPageView to use
new mapreduce api. (Amareshwari Sriramadasu via sharad)
HADOOP-5694. Change org.apache.hadoop.examples.dancing to use new
mapreduce api. (Amareshwari Sriramadasu via sharad)
HADOOP-5696. Change org.apache.hadoop.examples.Sort to use new
mapreduce api. (Amareshwari Sriramadasu via sharad)
HADOOP-5698. Change org.apache.hadoop.examples.MultiFileWordCount to
use new mapreduce api. (Amareshwari Sriramadasu via sharad)
HADOOP-5913. Provide ability to an administrator to stop and start
job queues. (Rahul Kumar Singh and Hemanth Yamijala via yhemanth)
MAPREDUCE-711. Removed Distributed Cache from Common, to move it
under Map/Reduce. (Vinod Kumar Vavilapalli via yhemanth)
HADOOP-6201. Change FileSystem::listStatus contract to throw
FileNotFoundException if the directory does not exist, rather than letting
this be implementation-specific. (Jakob Homan via cdouglas)
HADOOP-6230. Moved process tree and memory calculator related classes
from Common to Map/Reduce. (Vinod Kumar Vavilapalli via yhemanth)
HADOOP-6203. FsShell rm/rmr error message indicates exceeding Trash quota
and suggests using -skpTrash, when moving to trash fails.
(Boris Shkolnik via suresh)
HADOOP-6303. Eclipse .classpath template has outdated jar files and is
missing some new ones. (cos)
HADOOP-6396. Fix uninformative exception message when unable to parse
umask. (jghoman)
HADOOP-6299. Reimplement the UserGroupInformation to use the OS
specific and Kerberos JAAS login. (omalley)
HADOOP-6686. Remove redundant exception class name from the exception
message for the exceptions thrown at RPC client. (suresh)
HADOOP-6701. Fix incorrect exit codes returned from chmod, chown and chgrp
commands from FsShell. (Ravi Phulari via suresh)
NEW FEATURES
HADOOP-6332. Large-scale Automated Test Framework. (sharad, Sreekanth
Ramakrishnan, at all via cos)
HADOOP-4268. Change fsck to use ClientProtocol methods so that the
corresponding permission requirement for running the ClientProtocol
methods will be enforced. (szetszwo)
HADOOP-3953. Implement sticky bit for directories in HDFS. (Jakob Homan
via szetszwo)
HADOOP-4368. Implement df in FsShell to show the status of a FileSystem.
(Craig Macdonald via szetszwo)
HADOOP-3741. Add a web ui to the SecondaryNameNode for showing its status.
(szetszwo)
HADOOP-5018. Add pipelined writers to Chukwa. (Ari Rabkin via cdouglas)
HADOOP-5052. Add an example computing exact digits of pi using the
Bailey-Borwein-Plouffe algorithm. (Tsz Wo (Nicholas), SZE via cdouglas)
HADOOP-4927. Adds a generic wrapper around outputformat to allow creation of
output on demand (Jothi Padmanabhan via ddas)
HADOOP-5144. Add a new DFSAdmin command for changing the setting of restore
failed storage replicas in namenode. (Boris Shkolnik via szetszwo)
HADOOP-5258. Add a new DFSAdmin command to print a tree of the rack and
datanode topology as seen by the namenode. (Jakob Homan via szetszwo)
HADOOP-4756. A command line tool to access JMX properties on NameNode
and DataNode. (Boris Shkolnik via rangadi)
HADOOP-4539. Introduce backup node and checkpoint node. (shv)
HADOOP-5363. Add support for proxying connections to multiple clusters with
different versions to hdfsproxy. (Zhiyong Zhang via cdouglas)
HADOOP-5528. Add a configurable hash partitioner operating on ranges of
BinaryComparable keys. (Klaas Bosteels via shv)
HADOOP-5257. HDFS servers may start and stop external components through
a plugin interface. (Carlos Valiente via dhruba)
HADOOP-5450. Add application-specific data types to streaming's typed bytes
interface. (Klaas Bosteels via omalley)
HADOOP-5518. Add contrib/mrunit, a MapReduce unit test framework.
(Aaron Kimball via cutting)
HADOOP-5469. Add /metrics servlet to daemons, providing metrics
over HTTP as either text or JSON. (Philip Zeyliger via cutting)
HADOOP-5467. Introduce offline fsimage image viewer. (Jakob Homan via shv)
HADOOP-5752. Add a new hdfs image processor, Delimited, to oiv. (Jakob
Homan via szetszwo)
HADOOP-5266. Adds the capability to do mark/reset of the reduce values
iterator in the Context object API. (Jothi Padmanabhan via ddas)
HADOOP-5745. Allow setting the default value of maxRunningJobs for all
pools. (dhruba via matei)
HADOOP-5643. Adds a way to decommission TaskTrackers while the JobTracker
is running. (Amar Kamat via ddas)
HADOOP-4829. Allow FileSystem shutdown hook to be disabled.
(Todd Lipcon via tomwhite)
HADOOP-5815. Sqoop: A database import tool for Hadoop.
(Aaron Kimball via tomwhite)
HADOOP-4861. Add disk usage with human-readable size (-duh).
(Todd Lipcon via tomwhite)
HADOOP-5844. Use mysqldump when connecting to local mysql instance in Sqoop.
(Aaron Kimball via tomwhite)
HADOOP-5976. Add a new command, classpath, to the hadoop script. (Owen
O'Malley and Gary Murry via szetszwo)
HADOOP-6120. Add support for Avro specific and reflect data.
(sharad via cutting)
HADOOP-6226. Moves BoundedByteArrayOutputStream from the tfile package to
the io package and makes it available to other users (MAPREDUCE-318).
(Jothi Padmanabhan via ddas)
HADOOP-6105. Adds support for automatically handling deprecation of
configuration keys. (V.V.Chaitanya Krishna via yhemanth)
HADOOP-6235. Adds new method to FileSystem for clients to get server
defaults. (Kan Zhang via suresh)