Monday, September 5, 2016

Oracle full text search

Currently we use Compass indexes to speed up the full text search, I am not the fun of this solution.
There is a plan to migrate some functionality and I tried what Oracle Text is.


search

First I was surprised, to start with this feature you need only a few things:


  1. create a  TEXT index
  2. run your select to search

create index I_CONTENT_RAW_TEXT on T_CONTENT(RAW_TEXT_CLOB) indextype is ctxsys.context;

SELECT * FROM T_DOLA_CONTENT T 
WHERE CONTAINS(T.RAW_TEXT_CLOB, '%gäller%') >0

Fantastic!
You have to maintain your index of course:

  • update index
    you can run a little script in a DBMS_JOB: 

  CTX_DDL.SYNC_INDEX('I_CONTENT_RAW_TEXT', -- your index name here
                         '100M', -- memory count
                           NULL, -- param for partitioned idxes
                             2 -- parallel count
                               );
          • rebuild index
            CTX_DDL.OPTIMIZE_INDEX('I_CONTENT_RAW_TEXT', 'REBUILD');

          questions:


          • like search? it is working with the %
          • html encoded characters
          • the search is non-case-sensitive
          • special characters
          • glyphs


          Thursday, August 18, 2016

          get Oracle weblogic version

          You can find some details in the log: Dwls.home=/ec/local/weblogic/u000/app/owl1036/wlserver_10.3/server
           <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.6.0.160719 PSU Patch for BUG23094342

          But if it is not enough, you can create a little application to logout the weblogic.jar version number.


          public void getVersion() throws IOException{
          Manifest m = new JarFile(WEBLOGIC_JAR).getManifest();
          Attributes mainAttributes = m.getMainAttributes();
          for (Object key : mainAttributes.keySet()) {
          logger.info("Manifest key: " + key + " elem: " + mainAttributes.get(key));
          }
                 }

              and the result:
          2016-08-18 15:43:20 INFO  VersionPrinter:31 - Manifest key: Implementation-Vendor elem: BEA Systems
          2016-08-18 15:43:20 INFO  VersionPrinter:31 - Manifest key: Ant-Version elem: Apache Ant 1.7.1
          2016-08-18 15:43:20 INFO  VersionPrinter:31 - Manifest key: Implementation-Title elem: WebLogic Server 10.3.6.0  Tue Nov 15 08:52:36 PST 2011 1441050
          2016-08-18 15:43:20 INFO  VersionPrinter:31 - Manifest key: Implementation-Version elem: 10.3.6.0
          2016-08-18 15:43:20 INFO  VersionPrinter:31 - Manifest key: Class-Path elem: ../../../modules/features/weblogic.server.modules_10.3.6.0.jar schema/weblogic-domain-binding.jar schema/weblogic-domain-binding-compatibility.jar schema/diagnostics-binding.jar schema/diagnostics-image-binding.jar wlcipher.jar webservices.jar xmlx.jar ojdbc6.jar ons.jar ucp.jar aqapi.jar EccpressoAsn1.jar EccpressoCore.jar EccpressoJcae.jar mysql-connector-java -commercial-5.1.17-bin.jar  wlsqlserver.jar wldb2.jar wlsybase.jar wlinformix.jar fmwgenerictoken.jar wlw-langx.jar jcom.jar weblogic-L10N.jar
          2016-08-18 15:43:20 INFO  VersionPrinter:31 - Manifest key: Manifest-Version elem: 1.0
          2016-08-18 15:43:20 INFO  VersionPrinter:31 - Manifest key: Created-By elem: R28.2.0-79-146777-1.6.0_29-20111005-1808-windows-ia32 (Oracle Corporation)
          2016-08-18 15:43:20 INFO  VersionPrinter:31 - Manifest key: Main-Class elem: weblogic.Server

          Friday, July 15, 2016

          JSON de-serialization in Java - 2nd level

          You can have many problem when your system try to interpret incoming JSON content. It is a summary of problems and solutions with Jackson.

          Convert to Enum

          We like Enums in Java, because it is a terminated list of values. Sometimes you can find all the possible values of a property on the API, sometimes not. And of course you can get different undefined values, JSON is just a text.

          For preventing this issues you can use @JsonCreator and @JsonValue annotations.

          Let's create an Enum for the possible languages, for example we have a node:
          final HashMap<String, String> title = new HashMap<String, String>();
          We dont like the Strings, the first String is a language code, the second is the title of the content, like:

          {
              title : {
                        "EN": "title",
                        "FR": "title in FR"
              }
          }

          I created an Enum to restrict, reuse the languges.

          public enum MediaLanguage {
          INT, EN, FR, DE
          }

          INT means here it is international version. Dont care! And just for fun I recieved "INT/EN" once!

          Cool, lets see the annotations in an example:

          public enum MediaLanguage {
             INT, EN, FR, DE;

             public static List<String> names() {
          List<String> retval = new ArrayList<String>();
          for (MediaLanguage elem : MediaLanguage.values()) {
          retval.add(elem.name());
          }
          return retval;
             }

             public static boolean exist(String code) {
          return code != null && names().contains(code.toUpperCase());
             }

             @JsonCreator
             public static MediaLanguage getByJsonValue(String value) {
          if (exist(value)) {
          return MediaLanguage.valueOf(value.toUpperCase());
          }
          return EN;
             }

             @JsonValue
             public String toJsonValue() {
          return this.name();
             }
          }

          I created a getByJsonValue() function with @JsonCreator annotation. When Jackson tries to map it to MediaLanguage enum, he will call this function automatically to get the enum instance.
          First he will check, do we have this value in the list, and then return the instance. Just a comment, here you can use HasMaps to store the key-value pairs, if the enum element name is different then the JSON value.
          If we could not found, we return with a default value. I dont want to throw exception, I dont want lose data.

          I created a toJsonValue method with @JsonValue annotation. It will be called automativally by Jackson when he serialise the Enum to JSON string.

          This few functions are a little extensions in the Enum, but you can trust more and enjoy the type checks.

          Different JSON stucture


          Very annoying situation, if in one property we could receive different structure of JSON. For example:

                 "_id": 1234, 
                 "type": "PHOTO",
                 "cooperator": {
                        "AGENCY": ["Audiovisual Service  ", "Lorant"],
          "PHOTOGRAPHER": ["Benedek Dankahazi"]
          },

                  "_id": 1244,
                  "type": "VIDEO",
                  "cooperator": {
                         "I-12345-EN": {"PRODUCER": ["Audiovisual Service"]}
                  },

          It is not so funny. In the first case we can wrap the content with a HashMap<String, String> or later with a HashMap<MediaRole, String>:

          public class MediaDocumentJSON implements Serializable {
          private static final long serialVersionUID = 1L;
                  private Long _id;
                  private MediaType type;
                  private Map<MediaRole, String> cooperator = new HashMap<MediaRole, String>();
          }

          But the second case is not a hash map, we have an ID as a string and after that some roles in the HashMap<MediaRole, String>.

          The second difficulty is this key is always changing, like an id, we can not use @JsonProperty annotation.

          The only starting point is that we know when we receive the first and in which case the second one. So we can separate a little bit the interpretation logic.

          If you found a very dynamic node in the JSON like this, you can use the JsonNode type, and you will read this value later.

          public class MediaDocumentJSON implements Serializable {
          private static final long serialVersionUID = 1L;
                  private Long _id;
                  private MediaType type;
                  private JsonNode  cooperator;
          }

          Then we read this value in the process:
             if (PHOTO.equals(document.getType())) {
          return getPhotoCooperator(cooperatorJson);
             } else {
          return getAudioVideoCooperator(cooperatorJson);
             }

          The fist case you can cover with another class, or HashMap.
          protected static String getPhotoCooperator(String cooperatorJson) {
             try {
                CooperatorForPHOTOSJSON cooperator = mapper.readValue(nvl(cooperatorJson, ""), CooperatorForPHOTOSJSON.class);
                return StringUtils.join(cooperator.getPhotographer(), ",");
             } catch (Exception e) {
              return "";
             }
          }

          But for the second case we have to use the TypeFactory and the MapType:
          static ObjectMapper mapper = new ObjectMapper();
          static {
          TypeFactory typeFactory = mapper.getTypeFactory();
          MapType cooperatorVideoMapType = typeFactory.constructMapType(HashMap.class, String.class, CooperatorForVIDEOSJSON.class);
          }

          protected static String getAudioVideoCooperator(String cooperatorJson) {
          StringBuilder result = new StringBuilder();
          try {
          HashMap<String, CooperatorForVIDEOSJSON> cooperatorMap = mapper.readValue(nvl(cooperatorJson, ""), cooperatorVideoMapType);
          CooperatorForVIDEOSJSON cooperator = cooperatorMap.values().iterator().next();
          if (!cooperator.getDirector().isEmpty()) {
          result.append("Director: ").append(StringUtils.join(cooperator.getDirector(), ","));
          }
          } catch (Exception e) {
          return "";
          }
          return result.toString();
          }

          Good luck with the JSONs!

          Friday, May 20, 2016

          Mutable boolean

          Sometimes you have already a return value in your java function, but you need to update a boolean field from another function. If you try to give the reference of Java Boolean, it is not working.

          public void refresh(Object data){
             Boolean isChanged = Boolean.FALSE;
             updateFirstPart(data, isChanged);
             updateSecondPart(data, isChanged);
             if (isChanged) {
                saveData(data);
             }
          }

          protected void updateFirstPart(Object data, Boolean isChanged){
             if(conditions...){
                 data.setName("vmi");
                 isChanged = Boolean.TRUE;
             }
          }

          The data will be never saved, the isChanged filed keeps the false value.



          So for this we a Mutable Boolean type in Apache commons lang.

          import org.apache.commons.lang3.mutable.MutableBoolean;
          public void refresh(Object data){
             MutableBoolean isChanged = new MutableBoolean(false);
             updateFirstPart(data, isChanged);
             updateSecondPart(data, isChanged);
             if (isChanged.isTrue()) {
                saveData(data);
             }
          }

          protected void updateFirstPart(Object data, MutableBoolean isChanged){
             if(conditions...){
                 data.setName("vmi");
                 isChanged.setTrue();
             }
          }

          Tuesday, May 10, 2016

          String.startWith() challange, remove your BOM


          The system receives SOAP messages, they contain Base64 encoded XML content, we use as a String in the application layer and store as clob in the database.


          http://unsolublesugar.com/20120919/223812/

          We introduced a validation process to check this content, sometimes we have to use an ERROR status for example because the XML format is not good.

          This validation process first step to check the XML first row and should find: <!DOCTYPE .... But when we deploy to production we create always the ERROR status, the validation result is negative. Fortunately we store the XML content in the database and I found there the first character is very strange: \FFFE.

          OK, it is a BOM, I have to find a way to skip the BOM to able to test the doctype.

          We already use apache commons-io in the project and I found the BOMInputStream solution.

          Here you are my utility static method to convert the Base64 content to String and remove the possible BOM.

          import java.io.ByteArrayInputStream;
          import java.io.IOException;
          import java.io.StringWriter;

          import org.apache.commons.io.IOUtils;
          import org.apache.commons.io.input.BOMInputStream;


              public static String removeBOM(byte[] input) throws IOException {
                  if (input == null) {
                      return "";
                  }
                  StringWriter writer = new StringWriter();
                  IOUtils.copy(new BOMInputStream(new ByteArrayInputStream(input)), writer, UTF_8);
                  return writer.toString();
              }



          Friday, March 11, 2016

          Searching with subqueries and get last version using CriteriaBuilder

          I found a complex solution I share the details.

          The FindAll()

          You can put parameter using the Spring DAO findAll() method. So we have to create a Specification<T> object, which contain the conditions for the filtering. Of course we receive T entities, but what happen If we need to filter by children records, or these child tables are not linked in the T entity, or we also store different versions in the same table.

          The desired SELECT statement

          So I have TDocument entity, and a TDocuTranslationReq where we have versions. The link between the tables are the docuKy, it is a PK in the TDocument. We did not declared this relation in TDocument entity.
          The TDomain table a simple dictionary table, we store there for example the status records.

          The goal is to receive that TDocument records where the TDocuTranslatioReqLanguage status is a given status. It is a search functionality on the UI.

          SELECT *
            FROM T_DOCUMENT DOC
           WHERE EXISTS
           (SELECT null
                    FROM T_DOCU_TRANSREQ TR
                   INNER JOIN T_DOCU_TRANSREQ_LAN TR_LAN
                      ON TR.DOTR_KY = TR_LAN.DOTR_KY
                   CROSS JOIN T_DOMAIN STAT
                   WHERE TR_LAN.STATUS_DOMA_KY = STAT.DOMA_KY
                     AND DOC.DOCU_KY = TR.DOCU_KY
                     AND STAT.CODE_CD = 'EXECUTED'
                     AND TR.DOTR_KY =
                         (SELECT MAX(TR_LAST.DOTR_KY) FROM T_DOCU_TRANSREQ TR_LAST WHERE TR_LAST.DOCU_KY = TR.DOCU_KY))


           

          toPredicate()

          You can overwrite the toPredicate() function to get the right Specification<TDocument>.
          • we need a subquery to get all the linked TranslationRequest records
          • we have to filter this result with the dedicated status parameter in the TranslationRequestLanguages
          • we need a subquery to get the last version of the TranslationRequests

          Solution


              public static Specification<TDocument> translationRequestLanguageInStatusExists(final String status) {
                  return new Specification<TDocument>() {
                      @Override
                      public Predicate toPredicate(Root<TDocument> doc, CriteriaQuery<?> query, CriteriaBuilder cb) {
                          Subquery<TDocuTranslationRequest> subQuery = query.subquery(TDocuTranslationRequest.class);
                          Root<TDocuTranslationRequest> tr = subQuery.from(TDocuTranslationRequest.class);

                          ArrayList<Predicate> subQueryElements = new ArrayList<Predicate>();
                          subQueryElements.add(cb.equal(doc.get(TDocument_.docuKy), tr.get(TDocuTranslationRequest_.tDocument).get(TDocument_.docuKy)));
                          subQueryElements.add(cb.equal(tr.join(TDocuTranslationRequest_.tTransReqLanguages).get(TDocuTranslationRequestLanguage_.status).get(TDomain_.codeCd), status));

                          Subquery<Long> maxQuery = subQuery.subquery(Long.class);
                          Root<TDocuTranslationRequest> trMax = maxQuery.from(TDocuTranslationRequest.class);
                          maxQuery.select(cb.greatest(trMax.get(TDocuTranslationRequest_.dotrKy)));
                          ArrayList<Predicate> maxQueryElements = new ArrayList<Predicate>();
                          maxQueryElements.add(cb.equal(trMax.get(TDocuTranslationRequest_.tDocument).get(TDocument_.docuKy), tr.get(TDocuTranslationRequest_.tDocument).get(TDocument_.docuKy)));
                          maxQuery.where(maxQueryElements.toArray(new Predicate[] {}));
                          subQueryElements.add(cb.equal(tr.get(TDocuTranslationRequest_.dotrKy), maxQuery));

                          subQuery.select(tr);
                          subQuery.where(subQueryElements.toArray(new Predicate[] {}));
                          return cb.exists(subQuery);
                      }
                  };
              }

          • subQuery is a query 
            • on TDocuTranlationRequest entities (from clause), 
            • the conditions are collected in the subQueryelements list (where clause).
          • subQueryelements list contains all the conditions:
            • docuKy equijoin
            • status equijoin
            • last version equijoin
          • maxQuery is a subquery of the defined subQuery. 
            • It will return the primary key of the last version, the type is java Long and it . maxQuery.select(cb.greatest(trMax.get(TDocuTranslationRequest_.dotrKy))) (select clause)
            • from TDocuTranslationRequest (from clause)
            • The conditions are in the maxQueryelements list (where clause)
          • maxQueryElements list contains the conditions:
            • equijoin on docuKy

          Integration

          the integration of this seach parameter is additional. Of course there are other search parameters.
          specs = Specifications.where(specs).and(translationRequestLanguageInStatusExists(criteriaDTO.getTranslationRequestLanguageStatus().name()));
          List<TDocument> result = documentDAO.findAll(specs);

          Test

          Using dbUnit tests are very easy. I created sample data which are loaded to memory database.

              @Test
              public void shouldFindByRequestLangStatusByMax() throws Exception {
                  // given
                  Specifications<TDocument> spec = Specifications.where(translationRequestLanguageInStatusExists(STATUS));
                  // when
                  List<TDocument> result = documentDAO.findAll(spec);
                  // then
                  assertThat(result.size(), is(1));
                  TDocument doc = result.iterator().next();
                  assertThat(doc.getDocuKy(), is(10L));
              }

          Thursday, March 3, 2016

          If the developer lost the Admin rights on Windows

          Ok, sometimes you can not define your development environment. You have to use linux, you have to use windows. Or you don't have admin rights on your machine any more.

          I created a software collection, what to do in this case BEFORE.


          The conditions:

          • we have a c:\users\username user folder. we can store here files, we can not run executables.
          • we have preinstalled applications in the c:\Program Files
          • If you are lucky, you have a dedicated folder for the portable applications, lets call c:\PGM. You can execute the files here, but you don't have permission to modify, create.

           Recommended steps

          1. Ensure installed applications are in the c:\Program files
          2. Move you portable applications to PGM folder
          3. Reconfigure certain applications to reach config, ini, log, plugin etc files.
          4. hope

          Reconfiguration

          Ant

          • he has only a bat file which calls java, so you can put it into the c:\users\username folder. just update the ANT_HOME env variable.

          Eclipse

          • for the plug-ins and features he uses the c:\users\username\.eclipse folder
          • check the maven/User settings where the settings.xml and the repository are located

          Notepad++

          • if you have an old version he want to use the admin right when you restart after a plugin installation
            if you reinstall , he will ask to store plugins to the %APPDATA% folder

          Maven

          • settings.xml
          • .m2 folder for the repo, in default it is in the users folder
          • maven just a java, not executable, you can place it under the user folder. dont forget to update the MAVEN_HOME

          Total commander

          •  no problem with this, the FTP connections and settings are stored in the userfolder\Appdata

          Weblogic

          • you have to move you Oracle Weblogic domains to your user folders

          Monday, February 22, 2016

          Meetup: Big Data on Google Cloud

          We were invited to visit Google HQ in Bruxelles, Belgium. The building is very nice, easy to catch it by metro from Schuman, expect on the meetup day, there was a EU summit, everything was closed.



          The topics were interesting. The presenters shared their experience with google Cloud services.

          The Vente-Exclusive

          Alex von Boxel presented their solution which uses google cloug services. The Vente-Exclusive is a webshop, they collect users data and behaviour, and try to create a customised offer at nigth and send your email.

          Their concept is to collect 3TB data and load it to analyse automatically  with google cloud services.
          If you check the shared presentation you will see they use cloud storage, dataproc, bigquery, dataflow from google. They use also not google solutionsd, the data is coming from a MS SQL databse, and they use tableau for reporting.



          The BigQuery


          We got a good show, explanation about Google BigQuery service from Matthias Feys (Datatonic).



          DataFlow

          Alex came back for a quick presentation about DataFlow. you could use Google Cloud SDK to reach the services. We know to use this services is also a question of money.  Their google bill is around 4.000 EURO per month!




          Thursday, February 18, 2016

          How to mock new Date() calls in jUnit test

          You can find lots of solutions for this.
          I prefer to use Powermock, also because I use it for static code mocking.

          The production code:

              @Override
              @Transactional
              public Integer submitTranslationRequest(RequestInfo requestInfo, DocumentTranslationRequestDTO requestDTO) throws PRDException {

                   ...
                  requestDTO.setSentPoetryDate(new Date());
                  ....
              }

          Let's see the TEST code:

          @RunWith(PowerMockRunner.class)
          @PrepareForTest(DocumentTranslationRequestServiceImpl.class)
          public class DocumentTranslationRequestServiceImplTest {


              @InjectMocks
              private final DocumentTranslationRequestServiceImpl service = new DocumentTranslationRequestServiceImpl();


              @Test
              public void shouldSubmit() throws Exception {
                  // given

                  Date eventDate = new Date();
                  PowerMockito.whenNew(Date.class).withNoArguments().thenReturn(eventDate);

                  // when
                  Integer result = service.submitTranslationRequest(requestInfo, dto);

                  DocumentTranslationRequestDTO expectedDto = new DocumentTranslationRequestDTO();
                  expectedDto.setSentPoetryDate(eventDate);
                  // then
                  verify(translationRequestDataProvider).saveRequest(eq(expectedDto));

                  ...
            }
          }

          What we need:

                  <dependency>
                      <groupId>org.powermock</groupId>
                      <artifactId>powermock-api-mockito</artifactId>
                      <version>1.4.9</version>
                      <scope>test</scope>
                  </dependency>
                  <dependency>
                      <groupId>org.powermock</groupId>
                      <artifactId>powermock-module-junit4</artifactId>
                      <version>1.6.4</version>
                      <scope>test</scope>
                  </dependency>
                  <dependency>
                      <groupId>org.powermock</groupId>
                      <artifactId>powermock-core</artifactId>
                      <version>1.6.4</version>
                      <scope>test</scope>
                  </dependency>


          If you have problem, error which complains for the javaassist, check the different versions, maybe there is a conflict.