Hibernate Search doesn't index/reindex entities

Question

I'm trying to use Hibernate Search in my project (writing tests right now using junit + dbunit), but searching query doesn't return any results. I worked on this yesterday and got to conclusion that problem is Hibernate Search doesn't work well with dbunit @DatabaseSetup (similiar problem as in this unanswered question: link). I will go with more details, but firs things first, there is my entity class:

@Entity
@Indexed
public class User {
    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    @Column(name = "userId")
    private Long id;
    (...)
    @Column(nullable = false, unique = true)
    @Field(index = Index.YES, analyze=Analyze.YES, store=Store.NO)
    private String email;
    (...)
    @Column(nullable = false, unique = true)
    @Field(index = Index.YES, analyze=Analyze.YES, store=Store.NO)
    private String username;
    (...)
}

I save it to db by my DAO:

@Repository
public class UserDAOImpl implements UserDAO {

    @Autowired
    private SessionFactory sessionFactory;

    @Override
    public long save(User toSave) {

        return (Long) this.sessionFactory.getCurrentSession().save(toSave);
    }
(...)
}

This is code responsible for running lucene query:

@Override
    public List<User> searchByEmail(String email) throws InterruptedException {

        return generateHibernateSearchQueryFor("email", email).list();
    }

    private org.hibernate.Query generateHibernateSearchQueryFor(String field, String searchParam) {

        FullTextSession fullTextSession = Search.getFullTextSession(sessionFactory.getCurrentSession());
        QueryBuilder queryBuilder = fullTextSession.getSearchFactory().buildQueryBuilder().forEntity(User.class).get();

        org.apache.lucene.search.Query lQuery = queryBuilder.keyword().onFields(field).matching(searchParam).createQuery();
        org.hibernate.Query fullTextQuery = fullTextSession.createFullTextQuery(lQuery, User.class);

        return fullTextQuery;
    }

And this is how thing is configured in spring config:

<bean id="hibernate4AnnotatedSessionFactory"
          class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
        <property name="dataSource" ref="dataSource" />
        <property name="packagesToScan" value="me.ksiazka.model" />
        <property name="hibernateProperties">
            <props>
                <prop key="hibernate.dialect">org.hibernate.dialect.MySQLDialect</prop>
                <prop key="hibernate.show_sql">false</prop>
                <prop key="hibernate.cache.region.factory_class">org.hibernate.cache.ehcache.SingletonEhCacheRegionFactory</prop>
                <prop key="hibernate.cache.use_second_level_cache">true</prop>
                <prop key="hibernate.cache.use_query_cache">true</prop>
                <prop key="hibernate.hbm2ddl.auto">create</prop>

                <prop key="hibernate.search.default.directory_provider">filesystem</prop>
                <prop key="hibernate.search.default.indexBase">src/searching_indexes</prop>
            </props>
        </property>
    </bean>

Now how did I test at first. I configured my testing dataset with dbunit and created testing method like this:

    @Test
    @DatabaseSetup("classpath:/testsDataset.xml")
    public void searchByEmailTest() {

        User u1 = new User("Maciej", "Adamowicz", "k2", "[email protected]", "MacAda");
        userDAO.save(u1);

        List<User> u = null;
        try {
            //It worked at first - as new user was saved with hibernate he got his index in hibernate search indexes folder and searching found him.
            u = searchService.searchByEmail("[email protected]");
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        //I know there should be asserts, its just for simplification for need of moment.
        System.out.println(":: " + u.size());
        System.out.println(":: " + u.get(0).getName());
    }


    List<User> u2 = null;
    try {
        //[email protected] is in db - setted up by @DatabaseSetup
        u2 = searchService.searchByEmail("[email protected]");
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
    //This didnt work, rows putted into db by dbunit doesn't have indexes in my indexing folder.
    System.out.println(":: " + u2.size());
    System.out.println(":: " + u2.get(0).getName());
}

After looking Hibernate Search documentation i found fullTextSession.createIndexer().startAndWait(); method. I used it but it still doesn't work for rows from @DatabaseSetup. Anyway it worked with rows that I putted before test "by hand" with sql so I thought it is only problem with dbunit and just wrote setup with @Before:

@Before
    public void setupDatabase() {

        if(!doneBefore) {

            try {
                //It calls createIndexer().startAndWait() to make sure everything is indexed before test
                searchService.reindex();
            } catch (InterruptedException e) {    
                e.printStackTrace();
            }

            User u1 = new User("Maciej", "Adamowicz", "k2", "[email protected]", "MacAda");
            userDAO.save(u1);

            doneBefore = true;
        }

    }

And run this test:

    @Test
    public void searchByEmailTest() {

        List<User> u = null;
        try {
            u = searchService.searchByEmail("[email protected]");
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

        //Also asserts here, I know.
        System.out.println(":: " + u.size());
        System.out.println(":: " + u.get(0).getName());
    }

And it doesn't work althought data is saved by hibernate. I tried to find bug and reverted my code to eariel version where test was passing (the one with @DatabaseSetup but only for rows saved with my dao) and now this one doesn't pass too. I'm quite confused and out of ideas why it does not index new objects, not to say why it does not reindex all database when massive indexer is called. Any help will be appreciated.

EDIT:

After potential answers I did some more tests. In regards to fact that searching sometimes resulted with doubled or tripled rows I tried .purgeAll() and changing indexing provider to RAM to be sure that my indexes are clean when starting testing. It didn't change basicly anyting. To build my index I used .startAndWait() as mentioned before. Tried with building it "by hand" with .index() but I got some nested transactions problems when tried to use fullTextSession. Explicitly commiting transaction (or setting @Rollback(false) - tried both) doesn't work too. Everything I tried I found at Hibernate Search documentation - link. Indexing and searching work fine if I save something with DAO just before searching for it, but doing the same it @Before and then searching just doesn't work.

Answer 1

When I remember right, then Hibernate Search will update the index when you submit a transaction.

This is no problem for normal code, but in tests this behaviour can cause a problem, because a common pattern for tests is, that you start a transaction when you start the test, and at the end of the test you role the transaction back, but you never submit them.

To verify that this is the cause for your problem, create a test that start an explicite new transaction, modifiy something and then commit the transaction. Then after the commit check your hiberante search index.

Answer 2

As mentioned in this Hibernate Search doesn't index/reindex entities, you need to explicitly commit your transaction after saving data for indexing to occur. Indexing occurs on a post transaction synchronization (at least per default).

You can try to use the manual indexing API or the mass indexer. I am not sure why this did not work for you. I am also not sure how exactly @DatabaseSetup works and hooks into the JUnit life-cycle.

Regarding the triple results. You might be using a file system based index (used per default) which creates a file based Lucene index which is not cleaned up between test runs. Use a RAM index or make sure the file based index gets cleaned up.

It might help, if you share your Hibernate properties configuration.