Package mrv :: Module conf
[hide private]
[frames] | no frames]

Source Code for Module mrv.conf

   1  # -*- coding: utf-8 -*- 
   2  """ 
   3  Contains implementation of the configuration system allowing to flexibly control 
   4  the programs behaviour. 
   5   
   6   * read and write sections with key=value pairs from and to INI style file-like objects ! 
   7   * Wrappers for these file-like objects allow virtually any source for the operation 
   8   * configuration inheritance 
   9   * allow precise control over the inheritance behaviour and inheritance 
  10     defaults 
  11   * final results of the inheritance operation will be cached into the `ConfigManager` 
  12   * Environment Variables can serve as final instance to override values using the `DictConfigINIFile` 
  13   * Creation and Maintenance of individual configuration files as controlled by 
  14     submodules of the application 
  15   * These configuration go to a default location, or to the given file-like object 
  16   * embed more complex data to be read by specialised classes using URLs 
  17   * its safe and easy to write back possibly altered values even if complex inheritance 
  18     schemes are applied 
  19  """ 
  20  __docformat__ = "restructuredtext" 
  21   
  22  from ConfigParser import (      RawConfigParser, 
  23                                                          NoSectionError, 
  24                                                          NoOptionError, 
  25                                                          ParsingError) 
  26  from exc import MRVError 
  27  import copy 
  28  import re 
  29  import sys 
  30  import StringIO 
  31  import os 
  32  import logging 
  33  log = logging.getLogger("mrv.conf") 
  34   
  35  __all__ = ("ConfigParsingError", "ConfigParsingPropertyError", "DictToINIFile",  
  36             "ConfigAccessor", "ConfigManager", "ExtendedFileInterface", "ConfigFile",  
  37             "DictConfigINIFile", "ConfigStringIO", "ConfigChain", "BasicSet",  
  38             "Key", "Section", "PropertySection", "ConfigNode", "DiffData",  
  39             "DiffKey", "DiffSection", "ConfigDiffer") 
40 41 #{ Exceptions 42 ################################################################################ 43 -class ConfigParsingError(MRVError):
44 """ Indicates that the parsing failed """ 45 pass
46
47 -class ConfigParsingPropertyError(ConfigParsingError):
48 """ Indicates that the property-parsing encountered a problem """ 49 pass
50 #} End Exceptions
51 52 53 54 55 #{ INI File Converters 56 ################################################################################ 57 # Wrap arbitary sources and implicitly convert them to INI files when read 58 -class DictToINIFile(StringIO.StringIO):
59 """ Wraps a dictionary into an objects returning an INI file when read 60 61 This class can be used to make configuration information as supplied by os.environ 62 natively available to the configuration system 63 64 :note: writing back values to the object will not alter the original dict 65 :note: the current implementation caches the dict's INI representation, data 66 is not generated on demand 67 68 :note: implementation speed has been preferred over runtime speed """ 69 @classmethod
70 - def _checkstr(cls, string):
71 """ 72 :return: unaltered string if there was not issue 73 :raise ValueError: if string contains newline """ 74 if string.find('\n') != -1: 75 raise ValueError("Strings in INI files may not contain newline characters: %s" % string) 76 return string
77
78 - def __init__(self, option_dict, section = 'DEFAULT', description = ""):
79 """Initialize the file-like object 80 81 :param option_dict: dictionary with simple key-value pairs - the keys and 82 values must translate to meaningful strings ! Empty dicts are allowed 83 84 :param section: the parent section of the key-value pairs 85 :param description: will be used as comment directly below the section, it 86 must be a single line only 87 88 :raise ValueError: newlines are are generally not allowed and will cause a parsing error later on """ 89 StringIO.StringIO.__init__(self) 90 91 self.write('[' + str(section) + ']\n') 92 if len(description): 93 self.write('#'+ self._checkstr(description) + "\n") 94 for k in option_dict: 95 self.write(str(k) + " = " + str(option_dict[k]) + "\n") 96 97 # reset the file to the beginning 98 self.seek(0)
99
100 101 #} END GROUP 102 103 104 #{ Configuration Access 105 ################################################################################ 106 # Classes that allow direct access to the respective configuration 107 108 -class ConfigAccessor(object):
109 """Provides full access to the Configuration 110 111 **Differences to ConfigParser**: 112 As the functionality and featureset is very different from the original 113 ConfigParser implementation, this class does not support the interface directly. 114 It contains functions to create original ConfigParser able to fully write and alter 115 the contained data in an unchecked manner. 116 117 Additional Exceptions have been defined to cover extended functionality. 118 119 **Sources and Nodes**: 120 Each input providing configuration data is stored in a node. This node 121 knows about its writable state. Nodes that are not writable can be altered in memory, 122 but the changes cannot be written back to the source. 123 This does not impose a problem though as changes will be applied as long as there is 124 one writable node in the chain - due to the inheritance scheme applied by the configmanager, 125 the final configuration result will match the changes applied at runtime. 126 127 **Additional Information**: 128 The term configuration is rather complex though 129 configuration is based on an extended INI file format 130 its not fully compatible, but slightly more narrow regarding allowed input to support extended functionality 131 configuration is read from file-like objects 132 a list of file-like objects creates a configuration chain 133 keys have properties attached to them defining how they behave when being overridden 134 once all the INI configurations have been read and processed, one can access 135 the configuration as if it was just in one file. 136 Direct access is obtained though `Key` and `Section` objects 137 Keys and Sections have property attributes of type `Section` 138 Their keys and values are used to further define key merging behaviour for example 139 140 :note: The configaccessor should only be used in conjunction with the `ConfigManager`""" 141 __slots__ = "_configChain" 142
143 - def __init__(self):
144 """ Initialize instance variables """ 145 self._configChain = ConfigChain() # keeps configuration from different sources
146
147 - def __repr__(self):
148 stream = ConfigStringIO() 149 fca = self.flatten(stream) 150 fca.write(close_fp = False) 151 return stream.getvalue()
152 153 @classmethod
154 - def _isProperty(cls, propname):
155 """:return: true if propname appears to be an attribute """ 156 return propname.startswith('+')
157 158 @classmethod
159 - def _getNameTuple(cls, propname):
160 """:return: [sectionname,keyname], sectionname can be None""" 161 tokens = propname[1:].split(':') # cut initial + sign 162 163 if len(tokens) == 1: # no fully qualified name 164 tokens.insert(0, None) 165 return tokens
166
167 - def _parseProperties(self):
168 """Analyse the freshly parsed configuration chain and add the found properties 169 to the respective sections and keys 170 171 :note: we are userfriendly regarding the error handling - if there is an invlid 172 property, we warn and simply ignore it - for the system it will stay just a key and will 173 thus be written back to the file as required 174 175 :raise ConfigParsingPropertyError: """ 176 sectioniter = self._configChain.sectionIterator() 177 exc = ConfigParsingPropertyError() 178 for section in sectioniter: 179 if not self._isProperty(section.name): 180 continue 181 182 # handle attributes 183 propname = section.name 184 targetkeytokens = self._getNameTuple(propname) # fully qualified property name 185 186 # find all keys matching the keyname ! 187 keymatchtuples = self.keysByName(targetkeytokens[1]) 188 189 # SEARCH FOR KEYS primarily ! 190 propertytarget = None # will later be key or section 191 lenmatch = len(keymatchtuples) 192 excmessage = "" # keeps exc messages until we know whether to keep them or not 193 194 if lenmatch == 0: 195 excmessage += "Key '" + propname + "' referenced by property was not found\n" 196 # continue searching in sections 197 else: 198 # here it must be a key - failure leads to continuation 199 if targetkeytokens[0] != None: 200 # search the key matches for the right section 201 for fkey,fsection in keymatchtuples: 202 if not fsection.name == targetkeytokens[0]: continue 203 else: propertytarget = fkey 204 205 if propertytarget is None: 206 exc.message += ("Section '" + targetkeytokens[0] + "' of key '" + targetkeytokens[1] + 207 "' could not be found in " + str(lenmatch) + " candiate sections\n") 208 continue 209 else: 210 # section is not qualified - could be section or keyname 211 # prefer keynames 212 if lenmatch == 1: 213 propertytarget = keymatchtuples[0][0] # [(key,section)] 214 else: 215 excmessage += "Key for property section named '" + propname + "' was found in " + str(lenmatch) + " sections and needs to be qualified as in: 'sectionname:"+propname+"'\n" 216 # continue searching - perhaps we find a section that fits perfectly 217 218 219 # could be a section property 220 if propertytarget is None: 221 try: 222 propertytarget = self.section(targetkeytokens[1]) 223 except NoSectionError: 224 # nothing found - skip it 225 excmessage += "Property '" + propname + "' references unknown section or key\n" 226 227 # safety check 228 if propertytarget is None: 229 exc.message += excmessage 230 continue 231 232 propertytarget.properties.mergeWith(section) 233 234 # finally raise our report-exception if required 235 if len(exc.message): 236 raise exc
237 238 239 #{ IO Interface
240 - def readfp(self, filefporlist, close_fp = True):
241 """ Read the configuration from the file like object(s) representing INI files. 242 243 :note: This will overwrite and discard all existing configuration. 244 :param filefporlist: single file like object or list of such 245 246 :param close_fp: if True, the file-like object will be closed before the method returns, 247 but only for file-like objects that have actually been processed 248 249 :raise ConfigParsingError: """ 250 fileobjectlist = filefporlist 251 if not isinstance(fileobjectlist, (list,tuple)): 252 fileobjectlist = (filefporlist,) 253 254 # create one parser per file, append information to our configuration chain 255 tmpchain = ConfigChain() # to be stored later if we do not have an exception 256 257 for fp in fileobjectlist: 258 try: 259 node = ConfigNode(fp) 260 tmpchain.append(node) 261 node.parse() 262 finally: 263 if close_fp: 264 fp.close() 265 266 # keep the chain - no error so far 267 self._configChain = tmpchain 268 269 try: 270 self._parseProperties() 271 except ConfigParsingPropertyError: 272 self._configChain = ConfigChain() # undo changes and reraise 273 raise
274
275 - def write(self, close_fp=True):
276 """ Write current state back to files. 277 During initialization in `readfp`, `ExtendedFileInterface` objects have been passed in - these 278 will now be used to write back the current state of the configuration - the files will be 279 opened for writing if possible. 280 281 :param close_fp: close the file-object after writing to it 282 283 :return: list of names of files that have actually been written - as files can be read-only 284 this list might be smaller than the amount of nodes in the accessor. 285 """ 286 writtenFiles = list() 287 288 # for each node put the information into the parser and write to the node's 289 # file object after assuring it is opened 290 for cn in self._configChain: 291 try: 292 writtenFiles.append(cn.write(_FixedConfigParser(), close_fp=close_fp)) 293 except IOError: 294 pass 295 296 return writtenFiles
297 298 #} END GROUP 299 300 301 #{Transformations
302 - def flatten(self, fp):
303 """Copy all our members into a new ConfigAccessor which only has one node, instead of N nodes 304 305 By default, a configuration can be made up of several different sources that create a chain. 306 Each source can redefine and alter values previously defined by other sources. 307 308 A flattened chain though does only conist of one of such node containing concrete values that 309 can quickly be accessed. 310 311 Flattened configurations are provided by the `ConfigManager`. 312 313 :param fp: file-like object that will be used as storage once the configuration is written 314 :return: Flattened copy of self""" 315 # create config node 316 ca = ConfigAccessor() 317 ca._configChain.append(ConfigNode(fp)) 318 cn = ca._configChain[0] 319 320 # transfer copies of sections and keys - requires knowledge of internal 321 # data strudctures 322 count = 0 323 for mycn in self._configChain: 324 for mysection in mycn._sections: 325 section = cn.sectionDefault(mysection.name) 326 section.order = count 327 count += 1 328 section.mergeWith(mysection) 329 return ca
330 331 #} END GROUP 332 333 #{ Iterators
334 - def sectionIterator(self):
335 """:return: iterator returning all sections""" 336 return self._configChain.sectionIterator()
337
338 - def keyIterator(self):
339 """:return: iterator returning tuples of (`Key`,`Section`) pairs""" 340 return self._configChain.keyIterator()
341 342 #} END GROUP 343 344 #{ Utitlities
345 - def isEmpty(self):
346 """:return: True if the accessor does not stor information""" 347 if not self._configChain: 348 return True 349 350 for node in self._configChain: 351 if node.listSections(): 352 return False 353 # END for each node 354 return True
355 356 #} END GROUP 357 358 #{ General Access (disregarding writable state)
359 - def hasSection(self, name):
360 """:return: True if the given section exists""" 361 try: 362 self.section(name) 363 except NoSectionError: 364 return False 365 366 return True
367
368 - def section(self, section):
369 """ :return: first section with name 370 :note: as there might be several nodes defining the section for inheritance, 371 you might not get the desired results unless this config accessor acts on a 372 `flatten` ed list. 373 374 :raise NoSectionError: if the requested section name does not exist """ 375 for node in self._configChain: 376 if section in node._sections: 377 return node.section(section) 378 379 raise NoSectionError(section)
380
381 - def keyDefault(self, sectionname, keyname, value):
382 """Convenience Function: get key with keyname in first section with sectionname with the key's value being initialized to value if it did not exist. 383 384 :param sectionname: the name of the sectionname the key is supposed to be in - it will be created if needed 385 :param keyname: the name of the key you wish to find 386 :param value: the value you wish to receive as as default if the key has to be created. 387 It can be a list of values as well, basically anything that `Key` allows as value 388 389 :return: `Key`""" 390 return self.sectionDefault(sectionname).keyDefault(keyname, value)[0]
391
392 - def keysByName(self, name):
393 """:param name: the name of the key you wish to find 394 :return: List of (`Key`,`Section`) tuples of key(s) matching name found in section, or empty list""" 395 return list(self.iterateKeysByName(name))
396
397 - def iterateKeysByName(self, name):
398 """As `keysByName`, but returns an iterator instead""" 399 return self._configChain.iterateKeysByName(name)
400
401 - def get(self, key_id, default = None):
402 """Convenience function allowing to easily specify the key you wish to retrieve 403 with the option to provide a default value 404 405 :param key_id: string specifying a key, either as ``sectionname.keyname`` 406 or ``keyname``. 407 In case you specify a section, the key must reside in the given section, 408 if only a keyname is given, it may reside in any section 409 410 :param default: Default value to be given to a newly created key in case 411 there is no existing value. If None, the method may raise in case the given 412 key_id does not exist. 413 414 :return: `Key` instance whose value may be queried through its ``value`` or 415 ``values`` attributes""" 416 sid = None 417 kid = key_id 418 if '.' in key_id: 419 sid, kid = key_id.split('.', 1) 420 # END split key id into section and key 421 422 if sid is None: 423 keys = self.keysByName(kid) 424 try: 425 return keys[0][0] 426 except IndexError: 427 if default is None: 428 raise NoOptionError(kid, sid) 429 else: 430 for section in self.sectionIterator(): 431 return section.keyDefault(kid, default)[0] 432 # create default section 433 return self.sectionDefault('default').keyDefault(kid, default)[0] 434 # END default handling 435 # END option exception handling 436 else: 437 if default is None: 438 return self.section(sid).key(kid) 439 else: 440 return self.keyDefault(sid, kid, default)
441 # END default handling 442 # END has section handling 443 444 445 #} END GROUP 446 447 #{ Operators
448 - def __getitem__(self, key):
449 defaultvalue = None 450 if isinstance(key, tuple): 451 defaultvalue = key[1] 452 key = key[0] 453 # END default value handling 454 455 return self.get(key, defaultvalue)
456 457 #} END GROUP 458 459 460 #{ Structure Adjustments Respecting Writable State
461 - def sectionDefault(self, section):
462 """:return: section with given name. 463 :raise IOError: If section does not exist and it cannot be created as the configuration is readonly 464 :note: the section will be created if it does not yet exist 465 """ 466 try: 467 return self.section(section) 468 except: 469 pass 470 471 # find the first writable node and create the section there 472 for node in self._configChain: 473 if node.writable: 474 return node.sectionDefault(section) 475 476 # we did not find any writable node - fail 477 raise IOError("Could not find a single writable configuration file")
478
479 - def removeSection( self, name):
480 """Completely remove the given section name from all nodes in our configuration 481 482 :return: the number of nodes that did *not* allow the section to be removed as they are read-only, thus 483 0 will be returned if everything was alright""" 484 numReadonly = 0 485 for node in self._configChain: 486 if not node.hasSection(name): 487 continue 488 489 # can we write it ? 490 if not node._isWritable(): 491 numReadonly += 1 492 continue 493 494 node._sections.remove(name) 495 496 return numReadonly
497 498
499 - def mergeSection(self, section):
500 """Merge and/or add the given section into our chain of nodes. The first writable node will be used 501 502 :raise IOError: if no writable node was found 503 :return: name of the file source that has received the section""" 504 for node in self._configChain: 505 if node._isWritable(): 506 node.sectionDefault(str(section)).mergeWith(section) 507 return node._fp.name() 508 509 raise IOError("No writable section found for merge operation")
510
511 512 #} END GROUP 513 514 515 -class ConfigManager(object):
516 """ Cache Configurations for fast access and provide a convenient interface 517 518 The the ConfigAccessor has limited speed due to the hierarchical nature of 519 configuration chains. 520 The config manager flattens the chain providing fast access. Once it is being 521 deleted or if asked, it will find the differences between the fast cached 522 configuration and the original one, and apply the changes back to the original chain, 523 which will then write the changes back (if possible). 524 525 This class should be preferred over the direct congiguration accessor. 526 This class mimics the ConfigAccessor inteface as far as possible to improve ease of use. 527 Use self.config to directly access the configuration through the `ConfigAccessor` interface 528 529 To use this class, read a list of ini files and use configManager.config to access 530 the configuration. 531 532 For convenience, it will wire through all calls it cannot handle to its `ConfigAccessor` 533 stored at .config""" 534 535 __slots__ = ('__config', 'config', '_writeBackOnDestruction', '_closeFp') 536
537 - def __init__(self, filePointers=list(), write_back_on_desctruction=True, close_fp = True):
538 """Initialize the class with a list of Extended File Classes 539 540 :param filePointers: Point to the actual configuration to use 541 If not given, you have to call the `readfp` function with filePointers respectively 542 :type filePointers: `ExtendedFileInterface` 543 544 :param close_fp: if true, the files will be closed and can thus be changed. 545 This should be the default as files might be located on the network as shared resource 546 547 :param write_back_on_desctruction: if True, the config chain and possible 548 changes will be written once this instance is being deleted. If false, 549 the changes must explicitly be written back using the write method""" 550 self.__config = ConfigAccessor() 551 self.config = None # will be set later 552 self._writeBackOnDestruction = write_back_on_desctruction 553 self._closeFp = close_fp 554 555 self.readfp(filePointers, close_fp=close_fp)
556 557
558 - def __del__(self):
559 """ If we are supposed to write back the configuration, after merging 560 the differences back into the original configuration chain""" 561 if self._writeBackOnDestruction: 562 # might trow - python will automatically ignore these issues 563 self.write()
564
565 - def __getattr__(self, attr):
566 """Wire all queries we cannot handle to our config accessor""" 567 try: 568 return getattr(self.config, attr) 569 except Exception: 570 return object.__getattribute__(self, attr)
571 572 #{ IO Methods
573 - def write(self):
574 """ Write the possibly changed configuration back to its sources. 575 576 :raise IOError: if at least one node could not be properly written. 577 :raise ValueError: if instance is not properly initialized. 578 579 :note: It could be the case that all nodes are marked read-only and 580 thus cannot be written - this will also raise as the request to write 581 the changes could not be accomodated. 582 583 :return: the names of the files that have been written as string list""" 584 if self.config is None: 585 raise ValueError("Internal configuration does not exist") 586 587 588 # apply the changes done to self.config to the original configuration 589 try: 590 diff = ConfigDiffer(self.__config, self.config) 591 report = diff.applyTo(self.__config) 592 outwrittenfiles = self.__config.write(close_fp = self._closeFp) 593 return outwrittenfiles 594 except Exception,e: 595 log.error(str(e)) 596 raise
597 # for now we reraise 598 # TODO: raise a proper error here as mentioned in the docs 599 # raise IOError() 600
601 - def readfp(self, filefporlist, close_fp=True):
602 """ Read the configuration from the file pointers. 603 604 :raise ConfigParsingError: 605 :param filefporlist: single file like object or list of such 606 :return: the configuration that is meant to be used for accessing the configuration""" 607 self.__config.readfp(filefporlist, close_fp = close_fp) 608 609 # flatten the list and attach it 610 self.config = self.__config.flatten(ConfigStringIO()) 611 return self.config
612 613 #} End IO Methods 614 615 616 #{ Utilities 617 @classmethod
618 - def taggedFileDescriptors(cls, directories, taglist, pattern=None):
619 """Finds tagged configuration files in given directories and return them. 620 621 The files retrieved can be files like "file.ext" or can contain tags. Tags are '.' 622 separated files tags that are to be matched with the tags in taglist in order. 623 624 All tags must match to retrieve a filepointer to the respective file. 625 626 Example Usage: you could give two paths, one is a global one in a read-only location, 627 another is a local one in the user's home (where you might have precreated a file already). 628 629 The list of filepointers returned would be all matching files from the global path and 630 all matching files from the local one, sorted such that the file with the smallest amount 631 of tags come first, files with more tags (more specialized ones) will come after that. 632 633 If fed into the `readfp` or the `__init__` method, the individual file contents can override each other. 634 Once changes have been applied to the configuration, they can be written back to the writable 635 file pointers respectively. 636 637 :param directories: [string(path) ...] of directories to look in for files 638 :param taglist: [string(tag) ...] of tags, like a tag for the operating system, or the user name 639 :param pattern: simple fnmatch pattern as used for globs or a list of them (allowing to match several 640 different patterns at once) 641 """ 642 643 # get patterns 644 workpatterns = list() 645 if isinstance(pattern, (list , set)): 646 workpatterns.extend(pattern) 647 else: 648 workpatterns.append(pattern) 649 650 651 # GET ALL FILES IN THE GIVEN DIRECTORIES 652 ######################################## 653 from path import Path 654 matchedFiles = list() 655 for folder in directories: 656 for pattern in workpatterns: 657 matchedFiles.extend(Path(folder).files(pattern)) 658 # END for each pattern/glob 659 # END for each directory 660 661 # APPLY THE PATTERN SEARCH 662 ############################ 663 tagMatchList = list() 664 for taggedFile in sorted(matchedFiles): 665 filetags = os.path.split(taggedFile)[1].split('.')[1:-1] 666 667 # match the tags - take the file if all can be found 668 numMatched = 0 669 for tag in taglist: 670 if tag in filetags: 671 numMatched += 1 672 673 if numMatched == len(filetags): 674 tagMatchList.append((numMatched, taggedFile)) 675 676 # END for each tagged file 677 678 outDescriptors = list() 679 for numtags,taggedFile in sorted(tagMatchList): 680 outDescriptors.append(ConfigFile(taggedFile)) # just open for reading 681 return outDescriptors
682
683 #} end Utilities 684 685 686 #}END GROUP 687 688 689 690 691 #{Extended File Classes 692 693 -class ExtendedFileInterface(object):
694 """ Define additional methods required by the Configuration System 695 :warning: Additionally, readline and write must be supported - its not mentioned 696 here for reasons of speed 697 :note: override the methods with implementation""" 698 __slots__ = tuple() 699
700 - def isWritable(self):
701 """:return: True if the file can be written to """ 702 raise False
703
704 - def isClosed(self):
705 """:return: True if the file has been closed, and needs to be reopened for writing """ 706 raise NotImplementedError
707
708 - def name(self):
709 """ :return: a name for the file object """ 710 raise NotImplementedError
711
712 - def openForWriting(self):
713 """ Open the file to write to it 714 :raise IOError: on failure""" 715 raise NotImplementedError
716
717 718 -class ConfigFile(ExtendedFileInterface):
719 """ file object implementation of the ExtendedFileInterface""" 720 __slots__ = ['_writable', '_fp'] 721
722 - def __init__(self, *args, **kwargs):
723 """ Initialize our caching values - additional values will be passed to 'file' constructor""" 724 self._fp = file(*args, **kwargs) 725 self._writable = self._isWritable()
726
727 - def __getattr__(self, attr):
728 return getattr(self._fp, attr)
729
730 - def _modeSaysWritable(self):
731 return (self._fp.mode.find('w') != -1) or (self._fp.mode.find('a') != -1)
732
733 - def _isWritable(self):
734 """ Check whether the file is effectively writable by opening it for writing 735 :todo: evaluate the usage of stat instead - would be faster, but I do not know whether it works on NT with user rights etc.""" 736 if self._modeSaysWritable(): 737 return True 738 739 wasClosed = self._fp.closed 740 lastMode = self._fp.mode 741 pos = self.tell() 742 743 if not self._fp.closed: 744 self.close() 745 746 # open in write append mode 747 rval = True 748 try: 749 self._fp = file(self._fp.name, "a") 750 except IOError: 751 rval = False 752 753 # reset original state 754 if wasClosed: 755 self.close() 756 self._fp.mode = lastMode 757 else: 758 # reopen with changed mode 759 self._fp = file(self._fp.name, lastMode) 760 self.seek(pos) 761 # END check was closed 762 763 return rval
764
765 - def isWritable(self):
766 """:return: True if the file is truly writable""" 767 # return our cached value 768 return self._writable
769
770 - def isClosed(self):
771 return self._fp.closed
772
773 - def name(self):
774 return self._fp.name
775
776 - def openForWriting(self):
777 if self._fp.closed or not self._modeSaysWritable(): 778 self._fp = file(self._fp.name, 'w') 779 780 # update writable value cache 781 self._writable = self._isWritable()
782
783 -class DictConfigINIFile(DictToINIFile, ExtendedFileInterface):
784 """ dict file object implementation of ExtendedFileInterface """ 785 __slots__ = tuple() 786
787 - def isClosed(self):
788 return self.closed
789
790 - def name(self):
791 """ We do not have a real name """ 792 return 'DictConfigINIFile'
793
794 - def openForWriting(self):
795 """ We cannot be opened for writing, and are always read-only """ 796 raise IOError("DictINIFiles do not support writing")
797
798 799 -class ConfigStringIO(StringIO.StringIO, ExtendedFileInterface):
800 """ cStringIO object implementation of ExtendedFileInterface """ 801 __slots__ = tuple() 802
803 - def isWritable(self):
804 """ Once we are closed, we are not writable anymore """ 805 return not self.closed
806
807 - def isClosed(self):
808 return self.closed
809
810 - def name(self):
811 """ We do not have a real name """ 812 return 'ConfigStringIO'
813
814 - def openForWriting(self):
815 """ We if we are closed already, there is no way to reopen us """ 816 if self.closed: 817 raise IOError("cStringIO instances cannot be written once closed")
818
819 #} END extended file interface 820 821 822 #{ Utility Classes 823 824 -class _FixedConfigParser(RawConfigParser):
825 """The RawConfigParser stores options lowercase - but we do not want that 826 and keep the case - for this we just need to override a method""" 827 __slots__ = tuple() 828
829 - def optionxform(self, option):
830 return option
831
832 833 -class ConfigChain(list):
834 """ A chain of config nodes 835 836 This utility class keeps several `ConfigNode` objects, but can be operated 837 like any other list. 838 839 :note: this solution is mainly fast to implement, but a linked-list like 840 behaviour is intended """ 841 __slots__ = tuple() 842 843 #{ List Overridden Methods
844 - def __init__(self):
845 """ Assures we can only create plain instances """ 846 list.__init__(self)
847 848 @classmethod
849 - def _checktype(cls, node):
850 if not isinstance(node, ConfigNode): 851 raise TypeError("A ConfigNode instance is required", node)
852 853
854 - def append(self, node):
855 """ Append a `ConfigNode` """ 856 self._checktype(node) 857 list.append(self, node)
858 859
860 - def insert(self, node, index):
861 """ Insert L?{ConfigNode} before index """ 862 self._checktype(node) 863 list.insert(self, node, index)
864
865 - def extend(self, *args, **kwargs):
866 """ :raise NotImplementedError: """ 867 raise NotImplementedError
868
869 - def sort(self, *args, **kwargs):
870 """ :raise NotImplementedError: """ 871 raise NotImplementedError
872 #} END list overridden methodss 873 874 #{ Iterators
875 - def sectionIterator(self):
876 """:return: section iterator for whole configuration chain """ 877 return (section for node in self for section in node._sections)
878
879 - def keyIterator(self):
880 """:return: iterator returning tuples of (key,section) pairs""" 881 return ((key,section) for section in self.sectionIterator() for key in section)
882
883 - def iterateKeysByName(self, name):
884 """:param name: the name of the key you wish to find 885 :return: Iterator yielding (`Key`,`Section`) of key matching name found in section""" 886 # note: we do not use iterators as we want to use sets for faster search ! 887 return ((section.keys[name],section) for section in self.sectionIterator() if name in section.keys)
888 #} END ITERATORS
889 890 891 -def _checkString(string, re):
892 """Check the given string with given re for correctness 893 :param re: must match the whole string for success 894 :return: the passed in and stripped string 895 :raise ValueError: """ 896 string = string.strip() 897 # ALLOW EMPTY STRINGS AS VALUES 898 if not len(string): 899 return string 900 # raise ValueError("string must not be empty") 901 902 match = re.match(string) 903 if match is None or match.end() != len(string): 904 raise ValueError(_("'%s' Invalid Value Error") % string) 905 906 return string
907
908 -def _excmsgprefix(msg):
909 """ Put msg in front of current exception and reraise 910 :warning: use only within except blocks""" 911 exc = sys.exc_info()[1] 912 if hasattr(exc, 'message'): 913 exc.message = msg + exc.message
914
915 916 -class BasicSet(set):
917 """ Set with ability to return the key which matches the requested one 918 919 This functionality is the built-in in default STL sets, and I do not understand 920 why it is not provided here ! Of course I want to define custom objects with overridden 921 hash functions, put them into a set, and finally retrieve the same object again ! 922 923 :note: indexing a set is not the fastest because the matching key has to be searched. 924 Good news is that the actual 'is k in set' question can be answered quickly""" 925 __slots__ = tuple() 926
927 - def __getitem__(self, item):
928 # assure we have the item 929 if not item in self: 930 raise KeyError() 931 932 # find the actual keyitem 933 for key in iter(self): 934 if key == item: 935 return key 936 937 # should never come here ! 938 raise AssertionError("Should never have come here")
939
940 941 -class _PropertyHolderBase(object):
942 """Simple Base defining how to deal with properties 943 :note: to use this interface, the subclass must have a 'name' field""" 944 __slots__ = ('properties', 'name', 'order') 945
946 - def __init__(self, name, order):
947 # assure we do not get recursive here 948 self.properties = None 949 self.name = name 950 self.order = order 951 try: 952 if not isinstance(self, PropertySection): 953 self.properties = PropertySection("+" + self.name, self.order+1) # default is to write our properties after ourselves # will be created on demand to avoid recursion on creation 954 except: 955 pass
956 # END exception handling
957 958 959 -class Key(_PropertyHolderBase):
960 """ Key with an associated values and an optional set of propterties 961 962 :note: a key's value will be always be stripped if its a string 963 :note: a key's name will be stored stripped only, must not contain certain chars 964 :todo: add support for escpaing comas within quotes - currently it split at 965 comas, no matter what""" 966 __slots__ = ('_name', '_values', 'values') 967 validchars = r'[\w\(\)]' 968 _re_checkName = re.compile(validchars+r'+') # only word characters are allowed in key names, and paranthesis 969 _re_checkValue = re.compile(r'[^\n\t\r]+') # be as open as possible 970
971 - def __init__(self, name, value, order):
972 """ Basic Field Initialization 973 974 :param order: -1 = will be written to end of list, or to given position otherwise """ 975 self._name = '' 976 self._values = list() # value will always be stored as a list 977 self.values = value # store the value 978 _PropertyHolderBase.__init__(self, name, order)
979
980 - def __hash__(self):
981 return self._name.__hash__()
982
983 - def __eq__(self, other):
984 return self._name == str(other)
985
986 - def __repr__(self):
987 """ :return: ini string representation """ 988 return self._name + " = " + ','.join([str(val) for val in self._values])
989
990 - def __str__(self):
991 """ :return: key name """ 992 return self._name
993 994 @classmethod
995 - def _parseObject(cls, valuestr):
996 """ :return: int,float or str from valuestring """ 997 types = (long, float) 998 for numtype in types: 999 try: 1000 val = numtype(valuestr) 1001 1002 # truncated value ? 1003 if val != float(valuestr): 1004 continue 1005 1006 return val 1007 except (ValueError,TypeError): 1008 continue 1009 1010 if not isinstance(valuestr, basestring): 1011 raise TypeError("Invalid value type: only int, long, float and str are allowed", valuestr) 1012 1013 return _checkString(valuestr, cls._re_checkValue)
1014 1015
1016 - def _excPrependNameAndRaise(self):
1017 _excmsgprefix("Key = " + self._name + ": ") 1018 raise
1019
1020 - def _setName(self, name):
1021 """ Set the name 1022 :raise ValueError: incorrect name""" 1023 if not len(name): 1024 raise ValueError("Key names must not be empty") 1025 try: 1026 self._name = _checkString(name, self._re_checkName) 1027 except (TypeError,ValueError): 1028 self._excPrependNameAndRaise()
1029
1030 - def _getName(self):
1031 return self._name
1032
1033 - def _setValue(self, value):
1034 """:note: internally, we always store a list 1035 :raise TypeError: 1036 :raise ValueError: """ 1037 validvalues = value 1038 if not isinstance(value, (list, tuple)): 1039 validvalues = [value] 1040 1041 for i in xrange(0, len(validvalues)): 1042 try: 1043 validvalues[i] = self._parseObject(validvalues[i]) 1044 except (ValueError,TypeError): 1045 self._excPrependNameAndRaise() 1046 1047 # assure we have always a value - if we write zero values to file, we 1048 # throw a parse error - thus we may not tolerate empty values 1049 # NO: Allow that at runtime, simply drop these keys during file write 1050 # to be consistent with section handling 1051 self._values = validvalues
1052
1053 - def _getValue(self): return self._values
1054
1055 - def _getValueSingle(self): return self._values[0]
1056
1057 - def _addRemoveValue(self, value, mode):
1058 """Append or remove value to/from our value according to mode 1059 1060 :param mode: 0 = remove, 1 = add""" 1061 tmpvalues = value 1062 if not isinstance(value, (list,tuple)): 1063 tmpvalues = (value,) 1064 1065 finalvalues = self._values[:] 1066 if mode: 1067 finalvalues.extend(tmpvalues) 1068 else: 1069 for val in tmpvalues: 1070 if val in finalvalues: 1071 finalvalues.remove(val) 1072 1073 self.values = finalvalues
1074 1075 1076 #{ Utilities
1077 - def appendValue(self, value):
1078 """Append the given value or list of values to the list of current values 1079 1080 :param value: list, tuple or scalar value 1081 :todo: this implementation could be faster (costing more code)""" 1082 self._addRemoveValue(value, True)
1083
1084 - def removeValue(self, value):
1085 """remove the given value or list of values from the list of current values 1086 1087 :param value: list, tuple or scalar value 1088 :todo: this implementation could be faster (costing more code)""" 1089 self._addRemoveValue(value, False)
1090
1091 - def valueString(self):
1092 """ Convert our value to a string suitable for the INI format """ 1093 strtmp = [str(v) for v in self._values] 1094 return ','.join(strtmp)
1095
1096 - def mergeWith(self, otherkey):
1097 """Merge self with otherkey according to our properties 1098 1099 :note: self will be altered""" 1100 # merge properties 1101 if self.properties != None: 1102 self.properties.mergeWith(otherkey.properties) 1103 1104 #:todo: merge properly, default is setting the values 1105 self._values = otherkey._values[:]
1106 1107 #} END utilities 1108 1109 #{Properties 1110 name = property(_getName, _setName) 1111 """ Access the name of the key""" 1112 values = property(_getValue, _setValue) 1113 """ read: values of the key as list 1114 write: write single values or llist of values """ 1115 value = property(_getValueSingle, _setValue) 1116 """read: first value if the key's values 1117 write: same effect as write of 'values' """
1118 #} END properties
1119 1120 1121 -class Section(_PropertyHolderBase):
1122 """ Class defininig an indivual section of a configuration file including 1123 all its keys and section properties 1124 1125 :note: name will be stored stripped and must not contain certain chars """ 1126 __slots__ = ('_name', 'keys') 1127 _re_checkName = re.compile(r'\+?\w+(:' + Key.validchars+ r'+)?') 1128
1129 - def __iter__(self):
1130 """:return: key iterator""" 1131 return iter(self.keys)
1132
1133 - def __init__(self, name, order):
1134 """Basic Field Initialization 1135 1136 :param order: -1 = will be written to end of list, or to given position otherwise """ 1137 self._name = '' 1138 self.keys = BasicSet() 1139 _PropertyHolderBase.__init__(self, name, order)
1140
1141 - def __hash__(self):
1142 return self._name.__hash__()
1143
1144 - def __eq__(self, other):
1145 return self._name == str(other)
1146
1147 - def __str__(self):
1148 """ :return: section name """ 1149 return self._name 1150 1151 #def __getattr__(self, keyname): 1152 """:return: the key with the given name if it exists 1153 :raise NoOptionError: """ 1154 # return self.key(keyname) 1155 1156 #def __setattr__(self, keyname, value): 1157 """Assign the given value to the given key - it will be created if required"""
1158 # self.keyDefault(keyname, value).values = value 1159
1160 - def _excPrependNameAndRaise(self):
1161 _excmsgprefix("Section = " + self._name + ": ") 1162 raise
1163
1164 - def _setName(self, name):
1165 """:raise ValueError: if name contains invalid chars""" 1166 if not len(name): 1167 raise ValueError("Section names must not be empty") 1168 try: 1169 self._name = _checkString(name, Section._re_checkName) 1170 except (ValueError,TypeError): 1171 self._excPrependNameAndRaise()
1172
1173 - def _getName(self):
1174 return self._name
1175
1176 - def mergeWith(self, othersection):
1177 """Merge our section with othersection 1178 1179 :note:self will be altered""" 1180 # adjust name - the default name is mostly not going to work - property sections 1181 # possibly have non-qualified property names 1182 self.name = othersection.name 1183 1184 # merge properties 1185 if othersection.properties is not None: 1186 self.properties.mergeWith(othersection.properties) 1187 1188 for fkey in othersection.keys: 1189 key,created = self.keyDefault(fkey.name, 1) 1190 if created: 1191 key._values = list() # reset the value if key has been newly created 1192 1193 # merge the keys 1194 key.mergeWith(fkey)
1195 1196 #{ Properties 1197 name = property(_getName, _setName) 1198 #} 1199 1200 #{Key Access
1201 - def key(self, name):
1202 """:return: `Key` with name 1203 :raise NoOptionError: """ 1204 try: 1205 return self.keys[name] 1206 except KeyError: 1207 raise NoOptionError(name, self.name)
1208
1209 - def keyDefault(self, name, value):
1210 """:param value: anything supported by `setKey` 1211 :return: tuple: 0 = `Key` with name, create it if required with given value, 1 = true if newly created, false otherwise""" 1212 try: 1213 return (self.key(name), False) 1214 except NoOptionError: 1215 key = Key(name, value, -1) 1216 # set properties None if we are a propertysection ourselves 1217 if isinstance(self, PropertySection): 1218 key.properties = None 1219 self.keys.add(key) 1220 return (key, True)
1221
1222 - def setKey(self, name, value):
1223 """ Set the value to key with name, or create a new key with name and value 1224 1225 :param value: int, long, float, string or list of any of such 1226 :raise ValueError: if key has incorrect value 1227 """ 1228 k = self.keyDefault(name, value)[0] 1229 k.values = value
1230 #} END key acccess
1231 1232 1233 -class PropertySection(Section):
1234 """Define a section containing keys that make up properties of somethingI""" 1235 __slots__ = tuple()
1236
1237 1238 -class ConfigNode(object):
1239 """ Represents node in the configuration chain 1240 1241 It keeps information about the origin of the configuration and all its data. 1242 Additionally, it is aware of it being element of a chain, and can provide next 1243 and previous elements respectively """ 1244 #{Construction/Destruction 1245 __slots__ = ('_sections', '_fp')
1246 - def __init__(self, fp):
1247 """ Initialize Class Instance""" 1248 self._sections = BasicSet() # associate sections with key holders 1249 self._fp = fp # file-like object that we can read from and possibly write to
1250 #} 1251 1252
1253 - def _isWritable(self):
1254 return self._fp.isWritable()
1255 1256 #{Properties 1257 writable = property(_isWritable) # read-only attribute 1258 #} 1259
1260 - def _update(self, configparser):
1261 """ Update our data with data from configparser """ 1262 # first get all data 1263 snames = configparser.sections() 1264 validsections = list() 1265 for i in xrange(0, len(snames)): 1266 sname = snames[i] 1267 items = configparser.items(sname) 1268 section = self.sectionDefault(sname) 1269 section.order = i*2 # allows us to have ordering room to move items in - like properties 1270 for k,v in items: 1271 section.setKey(k, v.split(',')) 1272 validsections.append(section) 1273 1274 self._sections.update(set(validsections))
1275 1276
1277 - def parse(self):
1278 """ parse default INI information into the extended structure 1279 1280 Parse the given INI file using a _FixedConfigParser, convert all information in it 1281 into an internal format 1282 1283 :raise ConfigParsingError: """ 1284 rcp = _FixedConfigParser() 1285 try: 1286 rcp.readfp(self._fp) 1287 self._update(rcp) 1288 except (ValueError,TypeError,ParsingError): 1289 name = self._fp.name() 1290 exc = sys.exc_info()[1] 1291 # if error is ours, prepend filename 1292 if not isinstance(exc, ParsingError): 1293 _excmsgprefix("File: " + name + ": ") 1294 raise ConfigParsingError(str(exc))
1295 1296 # cache whether we can possibly write to that destination x 1297 1298 @classmethod
1299 - def _check_and_append(cls, sectionsforwriting, section):
1300 """Assure we ignore empty sections 1301 1302 :return: True if section has been appended, false otherwise""" 1303 if section is not None and len(section.keys): 1304 sectionsforwriting.append(section) 1305 return True 1306 return False
1307
1308 - def write(self, rcp, close_fp=True):
1309 """ Write our contents to our file-like object 1310 1311 :param rcp: RawConfigParser to use for writing 1312 :return: the name of the written file 1313 :raise IOError: if we are read-only""" 1314 if not self._fp.isWritable(): 1315 raise IOError(self._fp.name() + " is not writable") 1316 1317 sectionsforwriting = list() # keep sections - will be ordered later for actual writing operation 1318 for section in iter(self._sections): 1319 # skip 'old' property sections - they have been parsed to the 1320 # respective object (otherwise we get duplicate section errors of rawconfig parser) 1321 if ConfigAccessor._isProperty(section.name): 1322 continue 1323 1324 # append section and possibly property sectionss 1325 ConfigNode._check_and_append(sectionsforwriting, section) 1326 ConfigNode._check_and_append(sectionsforwriting, section.properties) 1327 1328 # append key sections 1329 # NOTE: we always use fully qualified property names if they have been 1330 # automatically generated 1331 # Autogenerated ones are not in the node's section list 1332 for key in section.keys: 1333 if ConfigNode._check_and_append(sectionsforwriting, key.properties): 1334 # autocreated ? 1335 if not key.properties in self._sections: 1336 key.properties.name = "+"+section.name+":"+key.name 1337 1338 1339 # sort list and add sorted list 1340 sectionsforwriting = sorted(sectionsforwriting, key=lambda x: -x.order) # inverse order 1341 1342 for section in sectionsforwriting: 1343 rcp.add_section(section.name) 1344 for key in section.keys: 1345 if len(key.values) == 0: 1346 continue 1347 rcp.set(section.name, key.name, key.valueString()) 1348 1349 1350 self._fp.openForWriting() 1351 rcp.write(self._fp) 1352 if close_fp: 1353 self._fp.close() 1354 1355 return self._fp.name()
1356 1357 #{Section Access 1358
1359 - def listSections(self):
1360 """ :return: list() with string names of available sections 1361 :todo: return an iterator instead""" 1362 out = list() 1363 for section in self._sections: out.append(str(section)) 1364 return out
1365 1366
1367 - def section(self, name):
1368 """:return: `Section` with name 1369 :raise NoSectionError: """ 1370 try: 1371 return self._sections[name] 1372 except KeyError: 1373 raise NoSectionError(name)
1374
1375 - def hasSection(self, name):
1376 """:return: True if the given section exists""" 1377 return name in self._sections
1378
1379 - def sectionDefault(self, name):
1380 """:return: `Section` with name, create it if required""" 1381 name = name.strip() 1382 try: 1383 return self.section(name) 1384 except NoSectionError: 1385 sectionclass = Section 1386 if ConfigAccessor._isProperty(name): 1387 sectionclass = PropertySection 1388 1389 section = sectionclass(name, -1) 1390 self._sections.add(section) 1391 return section
1392
1393 #} END section access 1394 #} END utility classes 1395 1396 1397 #{ Configuration Diffing Classes 1398 1399 -class DiffData(object):
1400 """ Struct keeping data about added, removed and/or changed data 1401 Subclasses should override some private methods to automatically utilize some 1402 basic functionality 1403 1404 Class instances define the following values: 1405 * ivar added: Copies of all the sections that are only in B (as they have been added to B) 1406 * ivar removed: Copies of all the sections that are only in A (as they have been removed from B) 1407 * ivar changed: Copies of all the sections that are in A and B, but with changed keys and/or properties""" 1408 __slots__ = ('added', 'removed', 'changed', 'unchanged','properties','name') 1409 1410
1411 - def __init__(self , A, B):
1412 """ Initialize this instance with the differences of B compared to A """ 1413 self.properties = None 1414 self.added = list() 1415 self.removed = list() 1416 self.changed = list() 1417 self.unchanged = list() 1418 self.name = '' 1419 self._populate(A, B)
1420
1421 - def toStr(self, typename):
1422 """ Convert own data representation to a string """ 1423 out = '' 1424 attrs = ['added','removed','changed','unchanged'] 1425 for attr in attrs: 1426 attrobj = getattr(self, attr) 1427 try: 1428 if len(attrobj) == 0: 1429 # out += "No " + attr + " " + typename + "(s) found\n" 1430 pass 1431 else: 1432 out += str(len(attrobj)) + " " + attr + " " + typename + "(s) found\n" 1433 if len(self.name): 1434 out += "In '" + self.name + "':\n" 1435 for item in attrobj: 1436 out += "'" + str(item) + "'\n" 1437 except: 1438 raise 1439 # out += attr + " " + typename + " is not set\n" 1440 1441 # append properties 1442 if self.properties is not None: 1443 out += "-- Properties --\n" 1444 out += str(self.properties) 1445 1446 return out
1447
1448 - def _populate(self, A, B):
1449 """ Should be implemented by subclass """ 1450 pass
1451
1452 - def hasDifferences(self):
1453 """:return: true if we have stored differences (A is not equal to B)""" 1454 return (len(self.added) or len(self.removed) or len (self.changed) or \ 1455 (self.properties is not None and self.properties.hasDifferences()))
1456
1457 1458 -class DiffKey(DiffData):
1459 """ Implements DiffData on Key level """ 1460 __slots__ = tuple() 1461
1462 - def __str__(self):
1463 return self.toStr("Key-Value")
1464 1465 @classmethod
1466 - def _subtractLists(cls, a, b):
1467 """Subtract the values of b from a, return the list with the differences""" 1468 acopy = a[:] 1469 for val in b: 1470 try: 1471 acopy.remove(val) 1472 except ValueError: 1473 pass 1474 1475 return acopy
1476 1477 @classmethod
1478 - def _matchLists(cls, a, b):
1479 """:return: list of values that are common to both lists""" 1480 badded = cls._subtractLists(b, a) 1481 return cls._subtractLists(b, badded)
1482
1483 - def _populate(self, A, B):
1484 """ Find added and removed key values 1485 1486 :note: currently the implementation is not index based, but set- and thus value based 1487 :note: changed has no meaning in this case and will always be empty """ 1488 1489 # compare based on string list, as this matches the actual representation in the file 1490 avals = frozenset(str(val) for val in A._values ) 1491 bvals = frozenset(str(val) for val in B._values ) 1492 # we store real 1493 self.added = self._subtractLists(B._values, A._values) 1494 self.removed = self._subtractLists(A._values, B._values) 1495 self.unchanged = self._subtractLists(B._values, self.added) # this gets the commonalities 1496 self.changed = list() # always empty - 1497 self.name = A.name 1498 # diff the properties 1499 if A.properties is not None: 1500 propdiff = DiffSection(A.properties, B.properties) 1501 self.properties = propdiff # attach propdiff no matter what
1502 1503
1504 - def applyTo(self, key):
1505 """Apply our changes to the given Key""" 1506 1507 # simply remove removed values 1508 for removedval in self.removed: 1509 try: 1510 key._values.remove(removedval) 1511 except ValueError: 1512 pass 1513 1514 # simply add added values 1515 key._values.extend(self.added) 1516 1517 # there are never changed values as this cannot be tracked 1518 # finally apply the properties if we have some 1519 if self.properties is not None: 1520 self.properties.applyTo(key.properties)
1521
1522 1523 -class DiffSection(DiffData):
1524 """ Implements DiffData on section level """ 1525 __slots__ = tuple() 1526
1527 - def __str__(self):
1528 return self.toStr("Key")
1529
1530 - def _populate(self, A, B ):
1531 """ Find the difference between the respective """ 1532 # get property diff if possible 1533 if A.properties is not None: 1534 propdiff = DiffSection(A.properties, B.properties) 1535 self.properties = propdiff # attach propdiff no matter what 1536 else: 1537 self.properties = None # leave it Nonw - one should simply not try to get propertydiffs of property diffs 1538 1539 self.added = list(copy.deepcopy(B.keys - A.keys)) 1540 self.removed = list(copy.deepcopy(A.keys - B.keys)) 1541 self.changed = list() 1542 self.unchanged = list() 1543 self.name = A.name 1544 # find and set changed keys 1545 common = A.keys & B.keys 1546 for key in common: 1547 akey = A.key(str(key)) 1548 bkey = B.key(str(key)) 1549 dkey = DiffKey(akey, bkey) 1550 1551 if dkey.hasDifferences(): self.changed.append(dkey) 1552 else: self.unchanged.append(key)
1553 1554 @classmethod
1555 - def _getNewKey(cls, section, keyname):
1556 """:return: key from section - either existing or properly initialized without default value""" 1557 key,created = section.keyDefault(keyname, "dummy") 1558 if created: key._values = list() # reset value if created to assure we have no dummy values in there 1559 return key
1560
1561 - def applyTo(self, targetSection):
1562 """Apply our changes to targetSection""" 1563 # properties may be None 1564 if targetSection is None: 1565 return 1566 1567 # add added keys - they could exist already, which is why they are being merged 1568 for addedkey in self.added: 1569 key = self._getNewKey(targetSection, addedkey.name) 1570 key.mergeWith(addedkey) 1571 1572 # remove moved keys - simply delete them from the list 1573 for removedkey in self.removed: 1574 if removedkey in targetSection.keys: 1575 targetSection.keys.remove(removedkey) 1576 1577 # handle changed keys - we will create a new key if this is required 1578 for changedKeyDiff in self.changed: 1579 key = self._getNewKey(targetSection, changedKeyDiff.name) 1580 changedKeyDiff.applyTo(key) 1581 1582 # apply section property diff 1583 if self.properties is not None: 1584 self.properties.applyTo(targetSection.properties)
1585
1586 1587 -class ConfigDiffer(DiffData):
1588 """Compares two configuration objects and allows retrieval of differences 1589 1590 Use this class to find added/removed sections or keys or differences in values 1591 and properties. 1592 1593 **Example Applicance**: 1594 Test use it to verify that reading and writing a (possibly) changed 1595 configuration has the expected results 1596 Programs interacting with the User by a GUI can easily determine whether 1597 the user has actually changed something, applying actions only if required 1598 alternatively, programs can simply be more efficient by acting only on 1599 items that actually changed 1600 1601 **Data Structure**: 1602 * every object in the diffing structure has a 'name' attribute 1603 1604 * ConfigDiffer.added|removed|unchanged: `Section` objects that have been added, removed 1605 or kept unchanged respectively 1606 1607 * ConfigDiffer.changed: `DiffSection` objects that indicate the changes in respective section 1608 1609 * DiffSection.added|removed|unchanged: `Key` objects that have been added, removed or kept unchanged respectively 1610 1611 * DiffSection.changed: `DiffKey` objects that indicate the changes in the repsective key 1612 1613 * DiffKey.added|removed: the key's values that have been added and/or removed respectively 1614 1615 * DiffKey.properties: see DiffSection.properties 1616 1617 * DiffSection.properties:None if this is a section diff, otherwise it contains a DiffSection with the respective differences 1618 """ 1619 __slots__ = tuple() 1620
1621 - def __str__(self):
1622 """ Print its own delta information - useful for debugging purposes """ 1623 return self.toStr('section')
1624 1625 @classmethod
1626 - def _getMergedSections(cls, configaccessor):
1627 """within config nodes, sections must be unique, between nodes, 1628 this is not the case - sets would simply drop keys with the same name 1629 leading to invalid results - thus we have to merge equally named sections 1630 1631 :return: BasicSet with merged sections 1632 :todo: make this algo work on sets instead of individual sections for performance""" 1633 sectionlist = list(configaccessor.sectionIterator()) 1634 if len(sectionlist) < 2: 1635 return BasicSet(sectionlist) 1636 1637 out = BasicSet() # need a basic set for indexing 1638 for section in sectionlist: 1639 # skip property sections - they have been parsed into properties, but are 1640 # still available as ordinary sections 1641 if ConfigAccessor._isProperty(section.name): 1642 continue 1643 1644 section_to_add = section 1645 if section in out: 1646 # get a copy of A and merge it with B 1647 # assure the merge works left-to-right - previous to current 1648 # NOTE: only the first copy makes sense - all the others that might follow are not required ... 1649 merge_section = copy.deepcopy(out[section]) # copy section and all keys - they will be altered 1650 merge_section.mergeWith(section) 1651 1652 #remove old and add copy 1653 out.remove(section) 1654 section_to_add = merge_section 1655 out.add(section_to_add) 1656 return out
1657
1658 - def _populate(self, A, B):
1659 """ Perform the acutal diffing operation to fill our data structures 1660 :note: this method directly accesses ConfigAccessors internal datastructures """ 1661 # diff sections - therefore we actually have to treat the chains 1662 # in a flattened manner 1663 # built section sets ! 1664 asections = self._getMergedSections(A) 1665 bsections = self._getMergedSections(B) 1666 # assure we do not work on references ! 1667 1668 # Deepcopy can be 0 in case we are shutting down - deepcopy goes down too early 1669 # for some reason 1670 assert copy.deepcopy is not None, "Deepcopy is not available" 1671 self.added = list(copy.deepcopy(bsections - asections)) 1672 self.removed = list(copy.deepcopy(asections - bsections)) 1673 self.changed = list() 1674 self.unchanged = list() 1675 self.name = '' 1676 common = asections & bsections # will be copied later later on key level 1677 1678 # get a deeper analysis of the common sections - added,removed,changed keys 1679 for section in common: 1680 # find out whether the section has changed 1681 asection = asections[section] 1682 bsection = bsections[section] 1683 dsection = DiffSection(asection, bsection) 1684 if dsection.hasDifferences(): 1685 self.changed.append(dsection) 1686 else: 1687 self.unchanged.append(asection)
1688 # END for each common section 1689
1690 - def applyTo(self, ca):
1691 """Apply the stored differences in this ConfigDiffer instance to the given ConfigAccessor 1692 1693 If our diff contains the changes of A to B, then applying 1694 ourselves to A would make A equal B. 1695 1696 :note: individual nodes reqpresenting an input source (like a file) 1697 can be marked read-only. This means they cannot be altered - thus it can 1698 be that section or key removal fails for them. Addition of elements normally 1699 works as long as there is one writable node. 1700 1701 :param ca: The configacceesor to apply our differences to 1702 :return: tuple of lists containing the sections that could not be added, removed or get 1703 their changes applied 1704 1705 - [0] = list of `Section` s failed to be added 1706 1707 - [1] = list of `Section` s failed to be removed 1708 1709 - [2] = list of `DiffSection` s failed to apply their changes """ 1710 1711 1712 # merge the added sections - only to the first we find 1713 rval = (list(),list(),list()) 1714 for addedsection in self.added: 1715 try: 1716 ca.mergeSection(addedsection) 1717 except IOError: 1718 rval[0].append(addedsection) 1719 1720 # remove removed sections - everywhere possible 1721 # This is because diffs will only be done on merged lists 1722 for removedsection in self.removed: 1723 numfailedremoved = ca.removeSection(removedsection.name) 1724 if numfailedremoved: 1725 rval[1].append(removedsection) 1726 1727 # handle the changed sections - here only keys or properties have changed 1728 # respectively 1729 for sectiondiff in self.changed: 1730 # note: changes may only be applied once ! The diff works only on 1731 # merged configuration chains - this means one secion only exists once 1732 # here we have an unmerged config chain, and to get consistent results, 1733 # the changes may only be applied to one section - we use the first we get 1734 try: 1735 targetSection = ca.sectionDefault(sectiondiff.name) 1736 sectiondiff.applyTo(targetSection) 1737 except IOError: 1738 rval[2].append(sectiondiff) 1739 1740 return rval
1741 1742 #} END configuration diffing classes 1743